00:00:43 Introduction of Pierre Pinson
00:01:25 Pierre Pinson’s background and his work in data-centric design engineering and forecasting.
00:02:20 How Pierre got into probabilistic forecasting and its applications in energy, logistics, and business analytics.
00:04:17 Assessing forecast quality, its importance in decision making, and how it connects with forecast value.
00:07:41 Initial reactions to probabilistic forecasting.
00:08:27 The problem of overconfidence.
00:10:00 Claude Bernard’s criticism of statistics and probabilities.
00:13:00 Determinism vs. stochastic behaviors in the world.
00:14:37 Bridging meteorology and business with probabilistic forecasting.
00:15:11 The importance of weather forecasting and cultural implications.
00:16:46 Explaining probabilities and understanding forecasts.
00:18:58 Challenges of information overload and decision-making.
00:20:31 Transforming probabilities into risk assessments.
00:22:14 Balancing automated decision-making and user trust.
00:23:36 Importance of meteorological forecasts in business and logistics.
00:25:01 Wind forecasts and their significance in the energy sector.
00:26:00 Weather data usage in power demand forecasting and supply chain situations.
00:30:25 Differences in applying probabilistic forecasting in meteorology and logistics contexts.
00:32:46 Discussing the challenges of translating complex probabilistic forecasts for clients.
00:33:32 Cost concerns of cloud computing and hosting large amounts of data.
00:35:02 Using two-dimensional histograms and their impact on memory and cost.
00:37:19 Teaching probabilistic forecasting and the challenges students face.
00:40:00 Making probabilistic forecasting easier and understanding model verification.
00:42:40 Inefficiency in process and transportation methods.
00:43:57 The challenge of removing uncertainties from the supply chain.
00:45:20 The cost of removing uncertainties and its impact on various industries.
00:47:00 The evolution of forecasting and its shift from applied mathematics to economics.
00:50:53 Convergence of different fields in forecasting and decision making under uncertainty.
00:52:30 Adapting the explanation of probabilistic forecasting for different backgrounds.
00:53:21 Applying probabilistic forecasting to various businesses and its benefits.
00:55:53 The appeal of visually interesting probabilistic forecasts and copyright infringement stories.
00:58:03 The limitations of pie charts in conveying information and their usage in pre-sale stages.
01:00:01 Embracing uncertainty in professional careers and understanding the probabilistic perspective.
01:02:23 Interdisciplinary approach and uncertainty in various industries.
01:04:27 Importance of education and how new generations impact the industry.
01:07:00 Probabilistic forecasting adoption curve in different fields.
01:08:33 Joannes’ view on a century-long time horizon for embracing uncertainty.
01:10:37 Challenges in adopting new ideas and the slow pace of change in some fields.
01:12:14 Importance of mathematics in forecasting technology.
01:13:26 Future advancements in forecasting science and technology.
Summary
In an interview with Conor Doherty, Joannes Vermorel, founder of Lokad, and Pierre Pinson, Chair of Data-centric Design Engineering at Imperial College London, discuss probabilistic forecasting and its applications in various fields. They emphasize the importance of understanding uncertainty in forecasting and the need for ongoing education in the area. All three agree that innovation happens faster than people can embrace it, and they encourage staying updated on new developments in the field and being prepared for the advancements yet to come.
Extended summary
In this interview, host Conor Doherty discusses probabilistic forecasting with guests Joannes Vermorel, the founder of Lokad, and Pierre Pinson, Chair of Data-centric Design Engineering at Imperial College London. Pinson has an extensive background in data-centric design engineering and has focused on various application areas, including energy and logistics. Vermorel, on the other hand, approached probabilistic forecasting from a supply chain perspective.
Pinson was initially interested in weather and renewable energy and was offered a PhD on forecasting for wind farms. He emphasizes the importance of understanding the uncertainty of a forecast and the potential range of outcomes. Vermorel’s journey towards probabilistic forecasting began with the realization that many supply chain forecasts were mostly zero. He found that while everything is possible, not everything is equally probable, and understanding the structure of forecast inaccuracies can be valuable.
Meteorology uses various metrics to assess forecast quality, such as the distance between predictions and actual outcomes, and the average of absolute differences between the two. However, these metrics may not always indicate whether a forecast is good or bad for a specific application. Vermorel adds that probabilistic forecasting can help provide an educated opinion on the realm of possibilities.
One challenge faced by those working with probabilistic forecasting is convincing others to accept and embrace the idea of quantifying uncertainty. People generally prefer deterministic forecasts due to cognitive biases that favor overconfidence. Probabilistic forecasts, however, provide a more transparent and fair representation of potential outcomes. Using probabilistic forecasts in decision-making can lead to better outcomes, but people need to be open to the idea of uncertainty.
Claude Bernard, a 19th-century French physiologist, argued against the use of statistics and probabilities in scientific experiments, suggesting that variability was a result of incomplete understanding or lazy science. However, Pinson believes that while deterministic approaches may work well for certain problems, the world is not fundamentally deterministic. Probabilistic forecasting is valuable for situations with inherent stochastic behavior and uncertainty.
One of the main challenges with probabilistic forecasting is the information overload. People already have a lot of information to process, and adding probabilistic data can make it even more difficult to make sense of everything. This can be especially true when dealing with large data sets, such as forecasting for millions of products in a supply chain.
To address this issue, some companies have turned to automated decision-making or risk assessments to help users make sense of probabilistic forecasts. By transforming probabilistic data into quantified risks, users can better understand the potential consequences of their decisions without being overwhelmed by the complexity of the data.
In the field of meteorology, probabilistic forecasting has proven useful for predicting variables like temperature, precipitation, wind speed, and solar radiation. These variables can have a significant impact on various aspects of daily life and business, such as energy production and consumption. In some cases, using weather data in supply chain forecasting can lead to more accurate predictions, particularly when dealing with sudden changes in weather patterns.
However, the interviewees also acknowledge that incorporating weather data into supply chain forecasting has been challenging, with few successful examples. One instance involved using weather data to improve power demand forecasts for an electrical provider in Europe. By incorporating weather data into their forecasts, the company was able to reduce inaccuracies caused by rapid changes in weather.
Vermorel shares his experiences with Lokad, which has achieved impressive accuracy in its forecasting models, despite their simplicity. One example he provides is a project with an ice cream vendor who wanted to forecast surges in demand based on weather conditions. Although the post-mortem analysis was successful in determining the reasons for increased sales, predicting demand proved to be more difficult due to the long lead times involved in the supply chain. Vermorel emphasizes that despite the challenges they faced, there is still potential for probabilistic forecasting to be successful in various industries.
Pinson discusses the differences between applying probabilistic forecasting in meteorological contexts versus logistics and business contexts. He explains that the main challenge is determining the right forecast product to be used as input for decision-making processes. He mentions that scenarios, intervals, and quantiles are some of the options that can be considered, but it ultimately depends on the specific needs of the client or customer.
Vermorel also highlights the importance of considering computing costs when implementing probabilistic forecasting techniques. In his experience, histograms and probability densities provide the most detailed information, but can be computationally expensive, especially when dealing with high-dimensional data. As a result, Lokad often employs a mix of techniques to keep costs manageable and calculations efficient.
When teaching students about probabilistic forecasting, Pinson finds that the biggest challenge is not convincing them of the concept’s merits, but rather helping them understand the practicalities of applying these techniques in real-world situations. Vermorel adds that it is crucial for practitioners to balance the theoretical aspects of probabilistic forecasting with the practical considerations of cost and computational efficiency.
Vermorel shares his struggles with teaching people who have already received an education from consultants advocating for lean movements and removing uncertainty from supply chains. He believes that some uncertainties can be removed, but others are acceptable and should be managed with proper tools.
Pinson emphasizes that removing uncertainty can be costly, and that it is better to accept and manage it wisely. He gives the example of renewable energy, where developing storage systems to handle an infinite amount of energy would be extremely expensive and not feasible. Instead, accepting uncertainty and forecasting can be more cost-effective and practical.
The discussion moves on to the historical and cultural aspects of forecasting, where people have always tried to live in a deterministic world and remove uncertainty. They also discuss the convergence of different fields, such as natural sciences, social sciences, and economics, in forecasting and decision-making under uncertainty.
Pinson talks about the challenges of teaching probabilistic forecasting to people with different backgrounds and the need for an abridged version for those who don’t have a strong mathematical background. He suggests starting with simple examples and gradually building up complexity, while emphasizing the importance of understanding the underlying principles and concepts.
Vermorel shares his experience with copyright infringement, as some of his company’s graphs were reused on LinkedIn without permission. However, these attractive graphs can capture the attention of potential customers and make the company appear more technologically advanced.
Pinson talks about how uncertainty is present in every aspect of our lives, and how understanding and managing it is crucial for professionals in various fields. Education plays a key role in promoting this understanding, as students who learn about probabilistic forecasting can bring these skills into the workforce and make a difference in their companies.
Pinson believes that the adoption of probabilistic forecasting will continue to grow across different industries, as more people become educated about it and companies look to each other for inspiration and ideas. He cites the shipping industry as an example of a field that has been slow to adopt probabilistic forecasting but is now looking to other fields for guidance on incorporating it into their operations.
Vermorel highlights the importance of understanding uncertainty in forecasting, citing the example of the Battle of the 19th century, where it took almost an entire century for people to admit that chemistry was relevant to medicine. He suggests that innovation happens faster than people can embrace it, and education plays a crucial role in this process. Vermorel also mentions Niels Bohr’s quote, “Science progresses one funeral at a time,” emphasizing the notion that significant progress may happen quickly, but understanding its implications takes seemingly forever.
Pinson discusses the applications of probabilistic forecasting in meteorology, mentioning that although the mathematics behind forecasting technology may not be the most relevant part of the discussion, it is essential to acknowledge the ongoing developments in applied mathematics. He explains that designing machinery to forecast millions of time series in parallel poses challenges, but researchers are continually developing new models and technologies for the future.
Both Vermorel and Pinson agree that there is still much progress to be made in forecasting and probabilistic forecasting, as well as the need for ongoing education and understanding of uncertainty. They encourage staying updated on new developments in the field and being prepared for the advancements yet to come.
Full Transcript
Conor Doherty: Welcome back to LokadTV! I’m your host, Conor, and as always, I’m joined by Lokad founder Joannes Vermorel. Joining us today is Pierre Pinson, he’s the editor-in-chief of the International Journal of Forecasting and Chief Scientist at Half Space. Today he’s going to talk to us about multiple interesting applications of probabilistic forecasting. Welcome to Lokad, nice to meet you.
Pierre Pinson: Thanks for having me today.
Conor Doherty: And thank you very much for joining us. We’re both very excited to have you. Now, Pierre, I actually gave a very brief introduction there; you actually have quite an expansive CV. So first of all, could you give everyone a glimpse into your background and what it is you do because I know you’re involved in multiple projects in multiple fields?
Pierre Pinson: Yes, so thanks very much. First of all, I’m a professor right now at Imperial College in London. I’m heading a chair that focuses on data-centric design engineering. As we have more and more data flowing these days, we have to generate value from data, and that’s the aim of my research and my teaching. Obviously, one of the most interesting and first applications we think of with data is forecasting, so it’s something I’ve been doing for the last 20 years, focusing on different types of application areas, mainly energy because there’s so much forecasting that’s relevant and needed for energy today, but also for logistics, business analytics, etc.
Conor Doherty: Loads of interesting fields there and a lot of overlap with what we want to do. So first of all, how did you get into probabilistic forecasting?
Pierre Pinson: Well, I never wanted to do probabilistic forecasting in the first place, I must confess. I was very interested in the weather and renewable energy, and I was offered to do a PhD on forecasting for renewables, for wind farms, you know, how much the wind farms are going to produce tomorrow. The thing is, a forecaster is always wrong, and my PhD supervisor said, “Well, we would like to know how wrong they are, but it should not just be one metric, right? Like a forecast is this good or this bad on average. We would like to know, right now, looking towards tomorrow, is my forecast going to be good or not?” And that’s how I drifted towards probabilistic forecasting, because then you have this idea, conditional to what I know today, projecting myself tomorrow, and how can I describe somehow the uncertainty of what may happen and tell maybe what’s the most likely outcome, what’s the expected outcome, and what would be maybe some interval for what may happen.
Conor Doherty: Well, that’s interesting, and I’m going to come to you, Joannes, in a moment about this, but when you talk about what makes a good forecast in meteorology, what metrics do you use to gauge the effectiveness? It can’t just be “well, it didn’t rain today, therefore it was 100% accurate.” What are the metrics for that?
Pierre Pinson: Yeah, that’s why it’s actually a science in itself, you know, how to assess a forecast and decide if it was a good forecast or not. In principle, when you have a forecast for a continuous variable, like wind speed or temperature, we think of forecast quality as related to a distance between what you predicted and what really happened. Then there are different ways to massage this distance. You can take a sum of squared distances or squared errors, you can look at the average of the absolute difference between the two, etc. So, there are a lot of metrics that tell how good the forecast is or is not. The problem is, it’s just a number. So, if I tell you on average my forecast is wrong by two, you might say, “Okay, great,” but does that make a good or bad forecast for my application? And that’s very often been the issue through the last decades of development in forecasting, is that we need to link the forecast quality, so how good you are in terms of this distance, to forecast value, how good it’s going to be if I use this forecast for my decision-making problems.
Conor Doherty: Any similarities there with how we approach it?
Joannes Vermorel: That’s very interesting because at Lokad, we came to probabilistic forecast through a completely different route, and what you describe is much closer to the educated opinion that I gained after a few years, as opposed to the starting point. My initial take was something much more mundane. It was the fact that, actually, we were forecasting many things in supply chains, and we would just forecast zeros. I’ve told this anecdote a couple of times because when we did our first attempts at forecasting things from sales for mini markets, where most products are sold like zero times per day on average, that’s the closest thing that you can get rounded to the closest integer. You have like 95% of the products in your typical mini market just having zero units being sold on any given day. And the problem was literally that we were having this problem, and so we started to have a forecasting bias that led us to quantiles. And then as we were toying with quantiles, we realized that we should probably have all the quantiles at once, and we went into probabilistic forecasts. But nowadays, when I have to explain why probabilistic forecasts matter, I think the way we approach it is that, yes, your forecast is desperately inaccurate, we know it. My forecast, in the case I’m a vendor, is desperately inaccurate, but this is not the same thing as saying I have no clue whatsoever about what those inaccuracies will be. Actually, I have a fairly educated opinion on the realm of possibilities. Everything is possible, but everything is not equally probable. In your educated opinion on the realm of possibilities, everything is possible but not equally probable, and there is a structure to analyzing error. Can you elaborate on that?
Pierre Pinson: The most bizarre and intriguing aspect to people is the idea that there is a structure to this analysis of error. People intuitively think uncertainty stems from not knowing things, and then you tell them there is a structure in what they don’t know. It sounds confusing. When I began advocating for probabilistic forecasts, the initial reaction I got was that no matter what happens, I would never be wrong because my forecast would always account for the chance of something happening. People saw it as the ultimate defense mechanism for the vendor.
Joannes Vermorel: It’s interesting to hear your anecdotes, but you could also see it from another perspective. There’s the “big mouse paradox” from psychology and marketing, where overconfident people in a room are given more credit, even if they may be wrong. Most people prefer a deterministic forecast because it gives them a feeling of confidence, even though they know it’s going to be wrong. Providing a probabilistic forecast is actually being more transparent and fair, but people have to accept that and go against their cognitive bias for determinism.
When we confess that we cannot be exact, but we can give a pretty good idea of the range of possibilities, we are being more transparent and possibly better in terms of forecast quality. The biggest issue for those of us working with probabilistic forecasting is getting people to accept the idea of embracing uncertainty and using it in decision making. It’s actually going to lead to better outcomes. This kind of determinism in your life is a big issue we have as people working with probabilistic forecasting. We have to somehow let people accept that they have to relax with this idea that they want this single point to be informative and true. They have to embrace the fact that as long as you can quantify uncertainty and use it in decision making, it’s going to be better. You can only do better.
Conor Doherty: What are your thoughts on this, Joannes?
Joannes Vermorel: It’s very interesting, this idea of admitting your own weakness. If you go back in time to Claude Bernard, who invented the control experiment, he made an entire case against the usage of statistics and probabilities. His point is actually very well made. He argued that if you have something that varies, it just means that you have a bad experiment, and you don’t control the stuff enough. He was in the field of medicine and said that if there is variability, there is a third variable that explains it. So, he was against the idea of using statistics and probabilities because, in his perspective, it was an admission of having an incomplete understanding of being a lazy scientist. You end up with these fancy statistics that are just an excuse for your own inadequacies. What is your perspective on this objection, Pierre?
Pierre Pinson: I agree that it’s true for certain set of problems where it’s only relying on the laws of physics, in a very controlled environment, and you think the deterministic approach should be good enough. You should not worry too much about all these uncertainties and having a probabilistic framework. But if you look at the more general case, it’s almost a philosophical statement about the world. Do we believe the world is fundamentally deterministic for everything that happens? Or is there actually some kind of stochastic behavior around us? You know, some stochastic rules, and so that makes the very basic idea of determinism not always applicable. We see that for the weather, and we’ve been trying to think, you know, if we have more and more measurements, if we are better with the laws of physics, somehow we should be able to look at it as a deterministic process and predict it. And that’s a good hope, but there’s been, I think, repeated experiments over the last 100 years where we realize eventually maybe not everything can be deterministic, and even these arguments that you mentioned, this incomplete knowledge, I think there’s so many things we need to model and predict where we will never have enough knowledge so that we can end up placing ourselves in a deterministic framework. It’s just not possible.
Conor Doherty: Well, if I could just jump in at that point because a question I wanted to ask a moment ago was it sort of just stitched together the idea of business and meteorology because there’s an interesting bridge between these two points, especially given your background here because, again, you have business and meteorological experience. So presumably, you found yourself in a room where you’ve applied probabilistic forecasting to a business problem, maybe encountered some degree of resistance, like “oh, I don’t want to use probabilistic forecasting, that’s the dark arts,” but that exact same person 10 seconds later takes out their iPhone and says, “oh, 60% chance it’ll rain later, better bring an umbrella.” How do you at that moment traverse that sort of cognitive dissonance?
Pierre Pinson: It’s a very good point, and there’s a cultural issue there. I think we’ve seen that, you mentioned the weather forecasting as an important field because it’s everywhere, everybody is using weather forecasting and is sensitive to it. It’s sensitive to this information, so it’s actually a field where it’s been seen that if we change the way we communicate the forecast information, there are things that work, there are things that don’t work. It’s difficult sometimes for people to really appraise what the information is about, but we find it useful whatever happens. We know we find it useful, but it’s a process, and I think the same process we have to go through it in different fields. So that can be in business, that can be in engineering problems, that can be in insurance, there can be so many things where actually, we as scientists or forecast providers, the industrial ecosystem, they have to contribute to change the culture so that people, clients, or users in general, they actually realize that we can think differently and that there are benefits.
Conor Doherty: Well, just to follow up there because you said that you found, if I understood correctly, that there were certain mechanisms for conveying information, meteorological information or meteorological forecasts, that resonated with people, and there were mechanisms that did not. So could you expand on that a bit more, please, or give perhaps an example so people can understand?
Pierre Pinson: So there have been different studies made by psychologists working with weather forecasters. For instance, if you were to make a statement talking about the probability of rain, like saying there’s a 60% chance of rain over London in the next two hours, people interpret that differently. Some people think it means it’s going to rain 60% of the time over London, while others believe it means there’s a 60% chance of rain at a given location in London. People have difficulty understanding what probability fundamentally means.
Joannes Vermorel: Yes, we’ve seen this issue as well when working with users or clients. There’s a significant amount of work that goes into developing probability forecast methodology and looking at how it could be used in practice. But there’s also a lot of work that has to go into helping people understand what the information really means and how it can affect their decision-making. The challenge is getting people to understand how to go from a probabilistic forecast to a decision that’s better than if they had used deterministic forecasts. If they don’t understand it, they won’t accept it.
Conor Doherty: How do you make it clear? Do you use a white box approach?
Joannes Vermorel: We’re doing something that is both similar and dissimilar at the same time. My struggle, coming from a supply chain background, is dealing with information overload. People already have too much information. Even deterministic forecasts can be overwhelming, as they’re often super aggregated and present various problems. When you move into the realm of probabilistic forecasting, it gets two orders of magnitude worse, with histograms for every data point and even more complexity if you consider high-dimensional probabilities.
Initially, we tried to improve visualization and other aspects, but in the end, we converged on a solution where we remove probabilities from the user perspective. We base decisions on probabilities, but we transform them into risk assessments expressed in currency. For example, we might tell a client that the risk of overstock is X amount, and the risk of stockout is Y amount. We quantify risk classes and perspectives, with the underlying probabilistic forecast being the “plumbing” for these assessments.
Of course, this isn’t the perfect solution, but it works for our clients. Sometimes, data scientist teams love working with probabilities, but supply chain experts who are less versed in probabilities find this approach more accessible. I would say the supply chain expert, who is incredibly competent in supply chain but not as versed in probabilities, finds it very difficult to pick their interest in that, just due to the information overload that demands. Very quickly, these managers have to assess if these histograms are even worth their time looking at. That’s a very tough selling point whenever you start talking to people who value their time highly.
Pierre Pinson: Agreed, I totally agree. There are very different trajectories. As you mentioned, there are different ways one could try to consider this issue of information overload. I totally agree with your strategy here. I think trying to have automated decision-making, or suggesting optimal decisions, having understood the cost-loss ratio of the user, etc., is something good. But again, they have to understand how you got there and why they should trust it in the first place, which is funny because when it was deterministic, they would trust it. And now that there’s the word probability, they don’t trust it anymore. But that’s another story. The beauty of having probabilistic information is that you can give this extra layer of very basic assessments, risk assessments, which is what they really want when they say they accept probabilistic. Please tell me what risk I’m exposed to. This is the simplest type of information you can provide that gives the benefits of probabilistic forecasting without this information overload that you mentioned. So, I actually think it’s a very good strategy.
Conor Doherty: Well, we’ve kind of started to merge the topics together, like meteorology and business. So, on this point, Pierre, because you have a lot of experience in both, what are some examples of key meteorological forecasts or meteorological data that you’ve taken and applied in a business or logistics context?
Pierre Pinson: The weather forecasting information that you may want to use as input to decision-making has very few variables that are extremely important, and then the importance of the variable decreases rapidly. The most important variables are temperature, which drives so many other processes in our lives, and then there’s precipitation. More recently, there’s wind because, in the old days, like 30 years ago, when weather forecasters would talk about wind, they would produce wind forecasts almost for fun, because no one cared too much. It’s just, is this going to be windy or not? And maybe if you do sailing, you’re a bit more interested. But today, because of energy applications, wind speed forecasts are extremely important because just a small forecast error in wind speed translates to huge forecast errors in the energy that’s going to be available tomorrow. So, think of a country like Denmark, where on average, half of the energy comes from the wind. It’s quite important to have good wind forecasts. These are the most relevant variables, and now it’s also going towards solar irradiance because of solar energy. But I would say these are the most important variables, and after that, in terms of impact, these weather variables are used everywhere today. I mean, when you look at the importance of weather forecasting and the quality of weather forecasts in our daily lives, both in a business context and our everyday life context, it’s extremely important.
Conor Doherty: Surely, in terms of lead times when it comes to goods being shipped from abroad, I mean, understanding what Pierre has just described, that must factor into probabilistic forecasting for supply chains, for example.
Joannes Vermorel: In the entire history of Lokad, I think we had only two occurrences where we actually managed to use weather data in supply chain situations. Again, that might be lack of talent, lack of dedication, or plenty of other things. But the bottom line was, to forecast, we had a large European electrical provider a decade ago, and we had a contract to improve their power demand forecast by taking into account the weather. And that, for me, is the only case where there was a very clear win in using weather data. It works, and the bottom line was we were looking at forecasts that were already quite aggregated, essentially by regions. So, the forecast even without the weather was already very accurate, I mean, like 2% accurate because they were very aggregated. But by the way, that was just power demand, not just, and power demand only for one day to the next. So, you were just looking 24 hours ahead, and the aggregated regions would be let’s say, a country like Belgium, or maybe France split into five areas, fairly high level.
Without weather, you would have a forecast, a time series forecast that was 2% accurate, and most of the inaccuracies were caused by rapid changes in weather. When it’s cold, it tends to remain cold, but then when suddenly you have a change in weather, you have this jump and you don’t see it. So, that was more like you had a forecast that was 0.5% accurate on average, but then you would accumulate maybe 5 or 6% inaccuracy on the day when the weather was changing. And by bringing weather in, they had already achieved something like 0.5% accuracy, and with Lokad, we achieved pretty much the same sort of accuracy but just with models that were actually much simpler and more manageable in terms of computer software. So, it was a specific type of undertaking.
That was the first thing where I’ve said that it does work, you know, really well. The second one was with FMCG brands who wanted to forecast essentially surges of demand using weather forecasts. Unfortunately, the results were mostly negative. What worked very well was, and I’m not going to give the name of the brand, but let’s say it was an ice cream vendor, and they just wanted to know what was working very well after the summer. The question was, did we sell more ice cream because it was very hot or because the marketing promotion was very good. As a post-mortem technique to explain the situation, it was working well. However, the problem when it comes to predicting is that the lead times involved in most supply chain situations are significant. For example, if you produce ice creams, you have to order raw materials and prepare your production schedule about six weeks in advance. At that point, the accuracy of weather forecasts tends to revert to seasonal averages, which are not sufficiently better than the seasonal average to change your decision. In our experience, it proved to be incredibly difficult, and we had few successes, but it was very educational in many ways.
Conor Doherty: Pierre, when it comes to applying probabilistic forecasting in a meteorological context, how does that translate to applying it in a logistics or business context? Are there similarities in terms of constraints or the process?
Pierre Pinson: One of the main issues that is key when we think of different applications, whether it’s meteorology or weather-sensitive sectors, is the type of forecast product that is useful as input to decision-making. Typically, weather forecasts use ensemble forecasts, which consist of multiple trajectories or potential futures. For example, the European Centre has 51 alternative scenarios. However, there are many types of decision processes where different forecast products are needed.
In trading, for instance, people prefer to use densities, which are full descriptions of the probability density function. Some people prefer intervals and pre-defined levels of confidence as input to decision-making. Others request specific quantiles based on their cost-loss considerations. So, the main difference I’ve seen between meteorology and other fields is that we have to spend a lot of time thinking about the right forecast product to use as input. We have to project ourselves into the world of our customers and find the best way to render the complex information from probabilistic forecasts into what’s most useful for them. So, the forecast should be useful for the clients, and your approach focuses on that as well. Can you elaborate more on the concerns with cloud computing costs and how it affects your work?
Joannes Vermorel: Yes, as an enterprise software vendor, one of the primary concerns is cloud computing cost, in a very mundane way. Just to give you some sort of scale, Lokad’s customer base is managing about a petabyte of data, and that’s what we are paying for right now to Microsoft, our cloud hosting provider. It is a good business for Microsoft and for Lokad, but there are costs involved. Most of what we are looking at is driven by the cost efficiency that we can have in terms of the computing hardware.
Histograms and probability densities are typically the best. They are super rich, super nice, and super easy to use. But the problem is that in one dimension, this is okay. You have a fixed cost, so you inflate the amount of data by, let’s say, 100 and you have a nice histogram. But then, if you go two-dimensional, because you want to have a matrix of probabilities, it gets more complicated. For example, you might want to look at the probabilities of having a specific demand for one product and the probabilities of having the same demand for another product, but jointly. The reason being that these products are competitors, and when demand surges for one product, it tends to be due to cannibalization from the other one. So you want probabilities that acknowledge the fact that most likely, if a product’s demand is going to surge, it means that another’s demand is going to diminish, and vice versa. Now, if you do it with a matrix, you have a two-dimensional histogram, and the memory required increases significantly. It gets worse as you go higher in dimensions, making things very costly.
Similarly, when you want to go into Monte Carlo style simulations, which are very good at dealing with high-dimensional stuff, the problem is that you have diminishing returns with many scenarios. You may need a lot of scenarios to even be able to observe a risk that is a little bit rare, like 10,000 instances. Most of our considerations are due to the fact that we need to keep computing costs manageable. It’s not just the cost that we pay to Microsoft, but also the fact that when you’re using something more complex, the calculations take longer, and people have to wait for the forecast to complete before they can go on with their tasks.
For deterministic time series techniques, especially the pre-machine learning ones that were used until the 90s, you can almost have them in real time, even if you have cyclicities and whatnot. They’re super snappy, and you can quickly get results with methods like ARIMA or exponential smoothing, all of those things can be done in real time, even if you have cyclicities and whatnot. But if you go for something like super fancy, like a very deep Deep Learning Network, this sort of thing can take hours to train, and so that for us, there is a lot of cost. Also, from our perspective, what tends to dominate is the practicality of it, and that’s a big concern.
Conor Doherty: Pierre, one of the things that you do is teach at Imperial College London, and you encounter students who come to your class to learn about probabilistic forecasting for the first time. With people who already have a background in math and are already sort of sold on the concept of embracing uncertainty, in your experience, what tends to be the biggest challenge that they have in terms of learning these skills, following your lead essentially?
Pierre Pinson: In terms of teaching forecasting, I’ve done that more while I was based in Denmark with a lot of focus on energy. I think the problems are always the same. One of the first problems is what we heard before; it’s about accepting it. Why would I go probabilistic in the first place? I have to say that normally, I spend quite a lot of time describing problems, decision-making problems, and showing the students that you can only make better decisions if you go probabilistic. It’s very important for a developer or user of a forecast, a client on both sides, to understand that you can only do better if you go probabilistic. It’s going to cost you something, but if you embrace it, it’s going to go better. It takes a lot of work to convince yourself and to realize and understand why. If you don’t understand why it would be better, maybe you’re going to have difficulties accepting it. So, we spend quite a lot of time doing that.
Then I want them to understand that it’s not difficult to produce these forecasts. You mentioned some of the classical models, but you can think of it even in terms of basic random variables. If you want to make a probabilistic forecast for something that’s Gaussian, when we’re doing the classical point forecast, we’re just predicting the mean. And now that we say we want to go probabilistic, we only need to predict the variance on top, and then we have a full probabilistic forecast. Even if you don’t want to go parametric, if you want to predict quantiles, you can actually use the same models as for the deterministic case, you just change the loss function in your training, and voila, you have a quantile forecast. One important aspect is that I teach the students that it is not something that is like orders of magnitude more complex to learn and handle.
The last part is verification because we also discussed earlier that some people have this idea that you take it easy now, right? Because if you go probabilistic, you could say whatever, and somehow you’re going to tell me afterward that you’re never wrong. But there are some very rigorous frameworks for verifying probabilistic forecasts and actually showing that those make sense, your probabilities are correct, and you’re trying to concentrate information, etc. These are the main blocks that I go through with my students, and my experience is that these are the foundations you need if you have to handle probabilistic forecasts after in your work.
Joannes Vermorel: My struggle is funny because you are probably privileged in a sense of having your students. My prospects typically have already received an education in the form of what consultants tell them. Let’s make enemies today. The problem is that there is an anti-advocacy, like the lean movement. The lean movement and the idea, for example, that we should waste less. I mean, as a general principle, yes, it is better to avoid waste. By definition, waste is something that is undesirable, so it’s tautological in nature in terms of the statement. Nobody is saying let’s produce waste for the sake of it, but that’s part of the lean manufacturing movement and lean supply chain movement. The problem with this line of thinking is that you end up with things like, for example, if you have wasteful processes like long lead times, you want to eliminate them. But at some point, you can have a process that is not necessarily very wasteful but very inefficient nonetheless, due to the fact that you have tried to compress your lead times as much as possible. For example, if you want to move stuff as fast as possible, an aircraft is best, but the efficiency of aircraft in terms of fuel is terrible compared to trains or cargo ships. So, in and of itself, going straight for the kill in terms of things like zero stock, zero delay, zero waste, which is the general perspective advocated for certain movements, is an attempt to remove entirely the uncertainty. If you have zero lead time, then suddenly why do you forecast? You only need to deal with whatever is right in front of you. If you have zero stock, why do you need to manage potential risk of overstock or whatever? So, my interesting take is that part of the challenge is that people who did not benefit in the early years of a course that was demonstrating the superiority of probabilistic thinking went through one or two, and sometimes three or four decades of opinionated advocacy from consultants to remove all sorts of uncertainties from their supply chain. Some of the uncertainties, I would say, are the accidental ones, where you just have uncertainties that emerge due to a crappy process – those, yes, you should remove. If people, for example, don’t have the proper skills and some people just do garbage, that’s not the sort of uncertainty you want. But you have other uncertainties, such as the fact that maybe cargo ships are a little bit slower due to the weather, so they’re not going to go at the same speed every single time. But it is perfectly acceptable to tolerate this uncertainty if you have the tools to deal with it in a way that is sensical.
Pierre Pinson: You’re right. Typically, we frame this issue as a matter of cost because, somehow, to remove uncertainties, there’s a cost. If you have an infinite amount of money that you’re ready to invest in removing all uncertainties, you could do it. But any kind of uncertainty, aside from the ones because you’re doing something wrong or some inefficiencies, any more fundamental uncertainty in your process that you want to remove, it’s going to cost you a lot, most likely. So, it’s a typical problem where you say, “Okay, great, you want to remove all uncertainties as if it came for free. If it came for free, we would have done it too.” So, this cost is something that we have to decide if we can bear. We have a parallel, which is very interesting in energy. For instance, I work a lot with renewable energy. If we were developing storage to have an infinite amount of energy we can store at any time and for as long as we want, somehow the problem would be over. Some people say we would not need to forecast, we would not need to worry. But developing and deploying batteries at this kind of scale
