00:00:07 Introduction of Warren Powell and the topic of the day.
00:00:36 Warren’s background and his work at Princeton and Casa Labs.
00:02:00 The subject of the discussion - uncertainty within supply chain management.
00:03:05 Comparison of truckload trucking and supply chain management.
00:06:00 The concept of sequential decision making in supply chain management.
00:09:01 KPIs, key performance indices, and running simulations in a supply chain.
00:10:00 The need for a decision-making rule to run simulations and evaluate company performance.
00:11:51 Using simulations to determine the best decision for a company.
00:13:03 The duality between probabilistic forecasting and a generative model.
00:15:17 The challenge of deploying these ideas in the field, the difficulty of reinforcement learning, and the potential of deep learning.
00:18:00 Discussion on the need to embrace complexity and build machine learning algorithms that can take policy-based decisions.
00:18:26 Explanation of how humans also use policies to make decisions.
00:19:37 Importance of computer simulations for supply chain management and their irreplaceable role.
00:22:17 Explanation of the four fundamental classes of methods for making decisions.
00:24:00 Criticism of current forecasting methods used by large fashion brands and the need to take into account cannibalization and substitution.
00:26:01 Discussion of the impact of discounts and sales on consumer behavior and how it affects businesses.
00:27:08 Comparison between using mathematical models and gut feeling to make business decisions.
00:29:44 Explanation of the importance of trust in policy-based forecasts.
00:30:32 Explanation of the need for knowledgeable people to understand the problem and to look at the right metrics.
33:37 Final thoughts on the future of supply chain management and the need for tooling and supply chain engineers.
Summary
The interview between Joannes Vermorel, founder of Lokad, and Warren Powell, Professor at Princeton University and co-founder of Optimal Dynamics, delves into the complexities and uncertainties of supply chain management and decision-making. The experts share their experiences in the field and offer insights on tackling these complexities through mathematical modeling and simulations. They emphasize the importance of policies and simulations for making strategic, tactical, and operational decisions in supply chain management, while highlighting the limitations of traditional forecasting methods. The interview concludes with a discussion about the future of policy-based methods in supply chain management and the need for skilled supply chain engineers.
Extended Summary
In the interview, Kieran Chandler hosts a discussion between Joannes Vermorel, founder of Lokad, and Warren Powell, Professor at Princeton University and co-founder of Optimal Dynamics. They address the complexities and uncertainties involved in supply chain management and decision-making.
Warren Powell shares his experience in the field, having established Castle Labs, a unique university-industry collaboration that tackles real-world problems. He discusses how his early work in truckload trucking exposed him to the challenges of planning for uncertain factors.
Joannes Vermorel elaborates on the core issue of sequential decision-making in supply chains, where present decisions are heavily influenced by future decisions. He compares this process to playing chess, where each move must be considered in the context of subsequent moves. Vermorel acknowledges that mathematically modeling these problems can be complex and puzzling.
Warren Powell explains that measuring the effectiveness of decision-making in supply chains involves using key performance indicators (KPIs) to assess the impact of decisions on cost and productivity. He suggests that simulations can help navigate the messy and unpredictable nature of supply chain management, as deterministic models may not provide accurate solutions.
The interview explores the challenges of managing uncertainties and making effective decisions in supply chains, where each decision is interconnected with future decisions. The experts discuss their experiences in the field and offer insights on tackling these complexities through mathematical modeling and simulations.
The conversation began with a comparison of supply chain optimization to playing chess against an unpredictable player, suggesting that the use of simulations can help make better decisions. Powell explained that policies or decision-making rules can be used alongside simulations to quickly evaluate company performance using a variety of metrics.
Vermorel agreed, emphasizing the importance of probabilistic forecasting and generative models, which can both be used for supply chain optimization. He discussed the duality between these two approaches and highlighted that the choice between them depends on the specific problem being addressed.
Both Vermorel and Powell agreed on the importance of using policies in conjunction with simulations to optimize supply chain decisions. Policies are abstract rules that can have parameters that can be learned and applied. Vermorel noted the difficulty of applying these concepts in real-world situations, as there have been decades of research with limited success in terms of numerical achievements.
Vermorel also pointed out that recent breakthroughs in deep learning and optimization methods, such as stochastic gradient descent, have improved the applicability of policy-based decision-making in complex environments. These techniques work well in noisy environments and with a large number of variables, making them suitable for real-world supply chain optimization problems.
Powell mentioned that humans also use policies or methods when making decisions and that there are four fundamental classes of methods for decision-making. He cited the example of Google Maps as a look-ahead policy, which could be useful in the context of long supply chains.
Powell emphasizes the necessity of simulations for making strategic, tactical, and operational decisions in supply chain management. Due to the lengthy time frames and complex nature of supply chains, trial and error isn’t a feasible method for testing ideas. Simulations, though imperfect, provide a better alternative. He highlights the importance of understanding the decisions being made, the evaluation metrics, the sources of uncertainty, and the decision-making process.
Vermorel, however, plays devil’s advocate by raising concerns about the credibility of numerical methods. He agrees that simulations are more effective than endless meetings, but he points out that many sophisticated mathematical models can be contextually naive. He cites the fashion industry as an example, where point-wise forecasts often ignore crucial factors like cannibalization and substitution. He stresses that gut feeling is usually more accurate when dealing with naive models.
Vermorel further argues that managers should take a more sympathetic approach to modeling by considering heuristics and embracing the problem. Meanwhile, Powell acknowledges that subtle modeling is essential for success, as simplistic models can overlook important factors, leading to potentially significant mistakes.
Both Vermorel and Powell agree that while computer simulations and advanced models are crucial for supply chain optimization, it is equally important to have a deep understanding of the problem at hand and to develop models that accurately reflect the complexities of the real-world supply chain.
The discussion revolves around the limitations of point forecasts and the advantages of policy-based forecasting methods.
The participants argue that traditional forecasting methods, which rely heavily on gut feelings and fail to account for the multitude of variables, often result in overstocking or understocking inventory. Point forecasts tend to produce very thin inventories, which they’ve learned is not optimal. Instead, they suggest that being realistic, intelligent, and asking the right questions will lead to better decision-making.
The challenge of getting people to trust and visualize the benefits of policy-based forecasting is also discussed. In the freight transportation industry, probabilistic forecasts are used to simulate various scenarios, which are then evaluated based on Key Performance Indicators (KPIs) to determine if they seem reasonable. This process helps to build trust in the method.
Both Vermorel and Powell emphasize the importance of having supply chain engineers who possess in-depth knowledge of the problem and programming skills. They agree that the best approach is to use a multitude of metrics to identify areas where decisions may be incorrect, costly, or inefficient. It’s important to focus on outliers, as they can often have significant consequences.
They address the limitations of average cost and point forecasts, emphasizing the need for supply chain engineers rather than data scientists or software engineers. They believe policy-based methods, which account for uncertainty and risk, will drive the future of supply chain management, aided by increasingly advanced computer capabilities.
The interview concludes with a discussion about the future of policy-based methods in supply chain management. Powell believes that point forecasts will become obsolete, as they do not accurately represent the real world. The advancements in computer technology and the increasing ability to handle uncertainty will make policy-based forecasting methods more effective and prevalent.
The interview highlights the limitations of traditional forecasting methods and emphasizes the advantages of policy-based forecasting, while stressing the importance of having skilled supply chain engineers and utilizing a variety of metrics for effective decision-making.
Full Transcript
Kieran Chandler: Today on Lokad TV, we’re delighted to be joined by Warren Powell, who’s going to discuss with us the difference between policy and point forecasts and how these can be used to optimize those catch-22 decisions. So Warren, thanks very much for joining us live from the States today, and as always, we like to start off by just learning a little bit about our guests. So perhaps you could just kick things off by telling us a bit about yourself.
Warren Powell: Well, first of all, thanks for inviting me on the program. I’ve really enjoyed doing what I’m doing and appreciate the opportunity to talk about it. I taught at Princeton for 39 years, and about 30 years ago, I set up this lab called Castle Labs. I was doing a lot of work with industry and got my start in freight transportation. Now, academics get a lot of money from governments and whatnot, but our major source of funding was from industry. I also realized that one of the weaknesses of government funding was that they don’t have any data; they don’t actually have a problem. So I developed this unique university-industry collaboration through Castle Labs, working with industry and tackling their problems. There was this early interest in using computers to help run companies more efficiently, so the lab went very well, it grew quickly, and I was lucky to get a large number of students. I think I ended up graduating about 60 grads and post-docs, and I will point to them as the major source of, I think, we wrote about 250 publications. That’s largely the work of the students. I started three consulting firms, the most recent is Optimal Dynamics that I’m still involved with. In fact, I retired last year to become more full-time involved in Optimal Dynamics. It’s a very exciting opportunity.
Kieran Chandler: Sounds great, and today our subject’s all about optimizing those decisions that you take within a supply chain. Perhaps you could just start this off by telling us what are the sorts of uncertainty that you can observe within supply chain management?
Warren Powell: Well, I do have to give one more bit of background, which is that my very earliest projects were in truckload trucking, and the one thing with truckload trucking and the introduction came from a large company called Schneider National. They already had computer models that would plan into the future deterministically, and they said, “Look, truckload trucking isn’t deterministic. We don’t know what’s going to happen tomorrow. We don’t know what’s going to happen today.” And I found that the academic community hadn’t learned about how to really model these problems and solve them on computers. So it launched several decades into me simply saying, “Okay, how do we even think about this problem?” because the academic community really didn’t have it worked out.
As I’ve moved from truckload trucking, which is large and complicated, to supply chain management, I’ve found that the latter is not anywhere near quite as complicated as the supply chain. So with truckload carriers, the major issue is whether the shipper will call on a load or not, how many loads do I have to move, and then there are a few other sources of noise like whether the driver shows up and whether he gets caught in traffic. It’s not quite nearly on the same scale as supply chains. So in supply chains, you get into it, and in truckload trucking, you might be making decisions a week or so in advance.
Kieran Chandler: Two in the future, most of it tends to be three or four days out. Supply chains can go out 100 days, 150 days. Ordering products from China can take multiple months. In those months, you can have major events, major storms, political problems, labor problems, and commodity shortages. A lot of this is actually happening to us today. The supplier has a lot of noise in terms of how long it takes for the manufacturer in China to actually build the product that you’re asking for. He may have to fire up a manufacturing line and get his parts and supplies lined up. Then you put it on the cargo ship, and the cargo ship can take 30 days, but it could take 35 or 36 depending on storms and weather. You can have port delays. When the product actually gets off the ship, it’s got unloading. You’ve got to put it on a railroad or a truck. Then when it finally arrives, you have to look at it and say, was the quality okay? I mean, it’s just this litany of different forms of uncertainty.
Kieran Chandler: Yeah, and Joannes, that’s what we’re going to discuss in a bit more detail today. What Warren was discussing there is the wide variation in time frames. Why is that interesting? What do those differences in time frames mean from maybe a more technical perspective?
Joannes Vermorel: I believe that Warren’s work is very interesting, but maybe for a slightly different aspect, which is the sequential decision-making process. The uncertainties are a bit of a technicality, but the core of the problem is to start even thinking about those sequential decisions that you make in sequence. The trick is that the future is shaping the past, which feels kind of wrong. The decision that you want to optimize right now actually depends on the decision that you will take later. Whether the decision you’re taking right now is a good one or not, it very much depends on the decision that will be taken later.
Joannes Vermorel: To clarify this sort of situation, let’s say you’re ordering from an overseas supplier with a minimal order quantity (MOQ), and you order tons of products from this supplier, and you have to reach a full container. Now, the thing is, you’re passing an order, and the container may contain hundreds of different products. Is it a good order? Well, it depends. It depends on when you will actually pass the order for the next containers. You see, the thing is, if you run out of a product just a few days after ordering your container, you may have a stock out for this product. Can you actually place another order from your supplier? No, not really, because this product on its own is only a tiny fraction of a whole container. So you’re stuck. You’re stuck with the fact that you’ve ordered a whole container, and you have to wait until you have an order capacity that is convenient, that is compatible with ordering a full container again.
Kieran Chandler: And so, the decision that you’re about to make, is it the right one? It depends on when you will make your next decision. The reality in supply chain and interacting in many areas is that, when you start thinking about what does it really mean to have a good decision, it has to be a decision that plays well with a decision that will be made later on. Just like if you’re playing chess, it’s not about did I make the correct move right now. It only makes sense to say this is a good move with regards to all the other moves that are about to be played. That’s what it means. And then the question is, suddenly the problem becomes very difficult to even approach mathematically because you’re thinking about, “Okay, I have a decision, and you can think I have different quantities I can choose.” But then you have some kind of a recursive perspective where you have to think about all the future decisions that have not been made, and you’re asking yourself, “I want to optimize this decision I have not taken yet, my decision, and yet I have to factor in future decisions that are not even made yet either.” You see, so you have this chicken and egg problem, and mathematically, it is difficult and puzzling. I think part of the process of the work of Warren Powell was to actually consolidate a corpus of mathematical framework and approaches so that you can even start thinking numerically about those problems in ways that are consistent. I really like that analogy of it being kind of similar to chess and very much dependent on what the other person is perhaps doing and what else is going on in the world.
Warren Powell: Well, all companies have ways of measuring performance. They call them the KPIs, the key performance indices. So they have all their metrics on cost, productivity. Companies have dozens and dozens, and sometimes hundreds of these metrics. You use these same metrics. What we tend to do is to run simulations. It would be nice if we had a deterministic world. Imagine Google Maps, where we sort of pretend that we know all the travel times, so we see the whole path to the destination. Supply chains are too messy. It goes too far, and there are too many random things going to happen, so it’s not one path; it’s many. So rather than looking in the future and thinking that you’re going to know exactly what’s going to happen, the chess analogy is a good one. But if you’re playing against an expert chess player, they tend to be very predictable. Imagine that they’re playing against a less expert chess player. That’s a little bit more random because they behave unpredictably. The problem actually gets much more complicated. But to keep it simple, imagine that the computer can just run simulations. As we get into the future, I need to have a rule, or I like to call it a policy, a method for making a decision that will tell me what decision I’m going to make no matter what happens. So, I can run these simulations, and anytime I get to a point, say a month out, and I need to make a decision, I have some rule. Computers can make these simulations very quickly. We can run 100 different scenarios into the future in parallel.
Kieran Chandler: Let’s start with how a company can use metrics to evaluate their performance. When it comes to making decisions, like whether to place an order and how big the order should be, how can companies simulate these decisions and their potential outcomes?
Warren Powell: To make these decisions, companies can use simulators to look at the metrics and determine the best decision for the present moment. We can use a rule for making a decision now, but we also need to use simulators to fine-tune that rule to ensure it works best, considering not only the present but also the future. For simpler problems, like inventory management, a simple order-up-to policy can work quite well. However, for more complex problems, such as when to place an order for a carload from China that will arrive 90 days from now, the decision needs to take into account factors like previous orders, known events like hurricanes, and other unknown factors. We can simulate these scenarios and evaluate the choices based on the best metrics for the future. Essentially, it’s about simulating and running your company the way you normally would and evaluating it accordingly.
Kieran Chandler: Johannes, what are your thoughts on this policy approach? Does simulation actually work in your eyes?
Joannes Vermorel: Yes, I very much agree that simulation works. When it comes to forecasting, the modern approach is to think about probabilistic forecasting. There is a duality between probabilistic forecasts and generative models. Probabilistic forecasts give you probabilities of certain futures, which can be sampled to draw examples of possible futures. Generative models, on the other hand, generate futures that, when averaged out, give you the probabilities of your holistic model. Essentially, these models are two different ways to look at the same thing. The choice between them is more a matter of technicalities and what is more appropriate for the numerical resolution of your problem.
Warren Powell: That’s right, and the key essence of the mathematical trick to optimize sequential decisions is to de-entangle the decisions of today and tomorrow. Instead of focusing on the decision itself, we need to learn the decision-making mechanism. This mechanism can have many parameters that can be optimized.
Kieran Chandler: Joannes, could you explain how policies can be tuned and their role in supply chain optimization?
Joannes Vermorel: A policy is fundamentally an abstract rule with parameters that can be learned one way or another. The idea is to confront your policies that generate decisions. To do this, you need a generative model, typically derived from a probabilistic forecast or other methods.
Kieran Chandler: How does thinking of simulation as a probabilistic forecast help in assessing the accuracy of a policy?
Joannes Vermorel: It’s interesting to view simulation as a probabilistic forecast because it allows you to consider accuracy. When people say they do simulation, the big question is whether the simulation provides an accurate depiction of the future. To answer this question, you need to adopt the probabilistic forecasting perspective, so you can assess if your simulator gives an accurate depiction of the possible futures for your company.
Kieran Chandler: Can you elaborate on the challenges of deploying these policies in the field and the progress made in computer science to overcome these challenges?
Joannes Vermorel: The problem becomes very much a matter of practicalities when it comes to deploying these ideas in the field. Learning policies has been an exceedingly difficult problem from a numerical perspective. Traditional reinforcement learning techniques were often successful only in toy setups, but struggled with problems involving thousands or millions of variables.
One of the side effects of the breakthrough in deep learning was the development of better mathematical optimization methods, such as stochastic gradient descent, which works well in noisy environments with transfer variables. These methods were not specifically designed for policy-making processes, but the progress in computer science has made them highly applicable to policy optimization in real-world, complex settings. This includes situations where simulations involve thousands or even hundreds of thousands of stock keeping units (SKUs) and span hundreds of days ahead.
Kieran Chandler: We have half a dozen of entangled uncertainties: uncertainty of demand, lead time, commodity price, cannibalization, competitor moves. They are not very complex, but they are just all entangled. So you need to be able to embrace this sort of ambient complexity so that your simulator is not super naive with regard to possible futures. Warren, would you agree with that? I mean, how would you go about building those machine learning algorithms to take those policy-based decisions?
Warren Powell: Well, first of all, let’s remind ourselves that humans are using policies also. Anytime a decision is made, you’re using a method. Let’s call that a policy, but it’s a method. Everybody’s using a method, so this isn’t actually new. What I’ve done is I said there are four fundamental classes of methods. The order up to is one of the four classes; it’s called a policy function evaluation.
Let’s take Google Maps as an example; that’s a look-ahead policy. In inventory planning, people have a tendency to use this simple policy function evaluation: when the order goes below something, order up to something. When you have these long supply chains, you really do need to think about the future. That’s a direct look-ahead, so now we’ve covered two of the four classes.
These four classes are just articulating what we’re doing anyway. Now, in order to find out what’s the best class, you have to simulate and see how it works over time. In a trucking company, I could come up with an idea, test it for a few weeks, and see if it works. You can’t do that with supply chains; it takes maybe close to a year to see if something’s going to work. The timeframes are just too long, and that’s why computer simulations are almost irreplaceable.
Without a computer, you’re just a bunch of people around a table arguing and saying, “Oh, well, I think this is better,” and somebody else thinks this is better. That’s what you see in the supply chain world today – a lot of people yammering and talking at each other, but nobody has any evidence. I have spent my career at Castle Labs building simulators for strategic planning, tactical planning, and simulators to see into the future to help me decide if I made the right decision right now.
Without simulations, what else are you going to do? I could try an idea, wait for three months, try a different idea, wait another three months, but of course, the next three months have nothing to do with the first three months. I don’t see any way to avoid having a computer to say, “Here’s one method of making decisions, here’s either a very different way or a slightly different way, another way of making decisions. Which one seems to work?” You run your simulations. Simulations are never perfect, but what’s better? Name me something that’s better.
The process is actually very simple to understand. One of the hardest things I find in supply chains is asking a supply chain professional what decisions they’re making. I can’t believe how many people will look at you blankly and say, “Well, I never really thought of it that way.” There’s more to supply chain than just ordering inventory; there’s a lot of other decisions involved.
Kieran Chandler: So, what are your metrics? It’s not just one; there’s a bunch of them. And what are your sources of uncertainty? We can have a long discussion about that, but you’re not going to list all of them. I’m sorry, these big supply chains you’re talking about involve global uncertainties. Who was going to anticipate that ship getting stuck in the Suez Canal or the COVID pandemic? But there’s a lot of noisy things you can anticipate, and we need to be aware that these things happen. So, once you’ve got the decisions you’re making, how you’re being evaluated, and the uncertainties, now you’re down to how to make the decision. And guess what, there are four classes of policies, all from the simplest order up to the most complicated full look ahead. You can look ahead deterministically, like Google Maps, or you can look ahead with uncertainty.
Warren Powell: When I talk to people in business and they use the word “forecast,” I have this feeling that every time I hear the word “forecast,” they mean point forecast. What I’ve loved about this show is that I don’t have to have that argument with you guys; you fully understand the need to think stochastically. You have to think about the uncertainty of your forecast, which means you have to think about rules for making decisions rather than the decision itself. And it’s all very straightforward. Yes, we need the computer. If you’re not going to use a computer, let me know what you’re going to do. If you’ve got a better idea, I can’t figure it out.
Kieran Chandler: Johannes, how much confidence can we have in these policy-based approaches? How solid are these mathematical modelizations?
Joannes Vermorel: First, I would agree that the counterpoint of having endless S&OP meetings is not very productive. However, one of the problems, and I believe it’s a valid criticism against numerical methods, is that, as Russell Ackoff said almost 40 years ago, we have methods that can be mathematically very sophisticated but contextually incredibly naive. So, the problem is that we need to start thinking about how much credibility we can have in those projections of the future. The reality is that there are many popular methods still widely used in large supply chains that are complete garbage. I understand the position of a manager looking at a fancy computer simulation that might be very sophisticated but that completely misses the point and relies on guesswork. To give you a more concrete example, I would say that probably almost every single large fashion brand nowadays does forecasts, most of them doing point-wise forecasts. So, you take your products and try to have weekly demand predictions. However, I would estimate that around 0% of those fashion brands are actually taking into account cannibalization and substitution.
Kieran Chandler: Welcome everyone to our interview today. We have two guests with us, Joannes Vermorel, the founder of Lokad, a software company that specializes in supply chain optimization, and Warren Powell, Professor at Princeton University, co-founder and Chief Analytics Officer at Optimal Dynamics. Thank you both for joining us. Let’s dive into the discussion.
Joannes Vermorel: When it comes to choosing a product to buy, you often just have a feeling and pick one that fits your taste or gut instinct. You can see that there are massive effects of substitution and cannibalization at play. For example, when I walk into a fashion store and see 20 different white shirts, they all kind of look the same to me. I end up favoring one over the other, but it’s not like I had an absolute idea of which barcode I wanted to take when I went into the store.
If you have a forecasting model that ignores something as massive as substitution, how much trust can you have in the mathematical model? I think there has been a lot of warranted skepticism because managers look at those fancy methods and ask, “Do you deal with something as basic as substitution?” If the answer is no, how can you trust the model?
You also need to consider the impact of discounts. If we start giving away large discounts at the end of the season, people will get used to the fact that our brand gives a lot of discounts, and they will wait for the next collection’s sale season to benefit from the discounts. People are smart, and they adapt.
So, when comparing a naive model to gut feeling, gut feeling is usually more correct. It is better to be approximately correct than exactly wrong. With proper care and understanding, it is possible to improve the model, but it requires a mechanical sympathy and embracing the problem to be approximately correct.
Warren Powell: I agree with your example, Joannes. There are two issues here: the randomness of choosing one shirt, which mathematical models can handle quite well, and the more subtle modeling of discounts and how the market reacts to them. A naive model could easily overlook the latter and suggest cutting prices, ignoring the fact that if you cut your price, the market becomes used to it. That’s a significant mistake.
People have a gut feeling, and more sophisticated models should be able to handle these subtleties. But a simple model won’t. I’ve spent years in the lab building models that companies were funding, and we’ve seen these sorts of mistakes occur when the models overlook crucial aspects of the problem.
Kieran Chandler: Would you mind discussing how you build confidence in your models, especially when working in freight transportation?
Warren Powell: It took us six or eight years to get a model that Norfolk Southern said they trusted. It was a lot of work. You can’t ignore the subtlety about the market getting used to something, it’s such an easy mistake to make. You need to calibrate the models, have good statistics, and knowledgeable people who will ask the right questions. However, constantly pulling things out of thin air is going to be tough. I think there’s too many variables for a human to consider, so one way to cover them is to order more inventory and hide it, which is expensive. If you use point forecasts, you tend to go with very thin inventories. We’re learning that’s not the right thing either.
Kieran Chandler: Warren, let’s go back to the idea of trust. When you’re building these policy-based forecasts, how do you get people to really visualize and buy into that vision? Because one of the real problems we had at Lokad was getting people to visualize what we were doing with probabilistic forecasts.
Warren Powell: In freight transportation, we do probabilistic forecasts of what shippers can do. We run simulations and look at what the trucks are actually doing to see if it seems reasonable. We put together metrics of how many times we’re covering loads or getting drivers home, and we look at the standard KPIs that any company would use. You can do your probabilistic forecast, run a thousand simulations, pull together your KPIs, and then ask if it looks reasonable. You’ve got to do a certain amount of trial and error, and smart people who understand the problem need to be involved. They’ve got to be looking at the KPIs and the right metrics to determine if it’s behaving correctly.
Kieran Chandler: Johannes, how would you compare your journey to Warren’s? It sounds like you’ve had some fairly similar experiences.
Joannes Vermorel: Yes, exactly. In Lokad, the people who have programming skills and in-depth knowledge of the supply chain problem itself are called supply chain scientists. It’s true that it cannot be approached from a monolithic single metric perspective, as this is usually very deceptive. When you do that, you end up with something that is very Kaggle-like, and not in a good way, where you micro-optimize one metric but then completely game it.
Kieran Chandler: Past its point of irrelevance, the approach is typically to have a lot of metrics that help you instrument your setup so that you can identify the areas where you’re doing something very wrong. Joannes, could you expand on this?
Joannes Vermorel: At Lokad, we call it experimental optimization. The idea is that you want to identify the situation where your numerical model that generates your policy is going to generate very poor decisions. This is typically the entry point to diagnose what’s wrong. The way to identify those outlier decisions that are missing something, like missing an elephant, is to look at the problem from many angles. These incorrect decisions can be very costly, but they are also infrequent. If you only look at your averages, you might not notice them because you’re looking at probabilistic metrics, which typically involve averages over many situations. The problem with averages is that they can hide something that only happens once in a thousand but multiplies your cost by a factor of 10. That’s why we try to instrument that.
The key is to have people that are supply chain engineers first and foremost, rather than being data scientists or software engineers. This brings me to the conclusion that you need some sort of tooling to operate with a decent level of productivity, but that’s a completely different problem.
Kieran Chandler: Thank you, Joannes. Warren, we’ll leave the last word to you. What are your hopes for the future? Can you see everyone using these policy-based methods one day?
Warren Powell: First of all, because everybody uses policy-based methods today, I think this is the way forward. The Google Maps of point forecasts, I’m sorry, but it’s just not going to survive. You have to understand, it was the beginning of my career in 1981.
Kieran Chandler: Welcome to Schneider National at the time, our largest truckload carrier in the United States and an early pioneer of analytics. In 1981, they already had computer models running, but they moved forward with point forecasts. They were the ones who came to me and said, “Warren, the world is stochastic. So now we move to the world of supply chain management. Point forecasts just aren’t going to work. I mean, maybe somebody in their mind thinks it’s the real world, but it’s not.” This notion of a policy-based approach describes how people make decisions.
Warren Powell: You can have simple policies like “order up to” or you can have look-ahead policies. Now, when people say “look ahead,” they tend to think deterministically. This is where things have to be new. You have to think “look ahead” with uncertainty, but the computers are now there. Back in the 1980s, the computers were an embarrassment. Now we sit on the cloud and we’ll run 50 scenarios in parallel, and instead of averages, we’ll assess how often a bad event happened to evaluate risk.
All the other issues Joannes raised are still valid. You still have to model the problem appropriately, but at some point, you’re going to have that crossover where what the computer is doing is still a lot better than what any human could do. I think that it’s tremendously exciting that the time is right. We’re already doing this in the truckload industry. I’ve written supply chain simulators, but mainly just as simulators. I think moving the simulator into the field, that says, “Okay, now I’m going to think about what might happen in the future to help me make a decision now,” is getting close.
Kieran Chandler: Okay, brilliant. I’m going to have to wrap up there, but gentlemen, thank you both for your time. So that’s everything for this week. Thanks very much for tuning in, and bye for now.
