00:00:13 Challenges of implementing proofs of concept in the supply chain.
00:01:00 When proofs of concept work well and how supply chain differs.
00:02:29 Supply chain as an open problem and difficulties in measuring its success.
00:03:31 Limitations of proofs of concept in the supply chain industry.
00:06:30 Importance of a holistic approach and lead times in supply chain optimization.
00:08:00 Importance of focusing on non-changing aspects of an initiative.
00:08:54 Difficulty of gathering data in proofs-of-concept.
00:11:55 Problems of simplifying an initiative into just forecasting.
00:13:54 The limitations of time-series forecasts.
00:15:17 Painful experiences with proofs-of-concept and their shortcomings.
00:17:36 Alternatives to POCs in assessing software vendors for supply chain solutions.
00:20:25 The future of POCs in the supply chain industry and their limitations.
00:21:31 Importance of taking a holistic approach to supply chain optimization.
00:22:52 Assessing vendor performance on a weekly basis instead of relying on POCs.
Summary
In an interview, Joannes Vermorel, Lokad founder, and host Kieran Chandler discuss supply chain optimization and the limitations of proofs of concept (POCs). Vermorel explains why Lokad often declines POC requests, citing harm to both parties. He believes POCs work best for narrowly defined problems like choosing an email client but fall short in complex, transformative processes like supply chain optimization. Supply chains involve multiple parties and present a distributed challenge that POCs often oversimplify. He advises focusing on stable aspects and data-driven, holistic perspectives instead. Vermorel also suggests regularly assessing vendor performance and ending relationships that do not show satisfactory progress.
Extended Summary
In the interview, Kieran Chandler, the host, and Joannes Vermorel, founder of Lokad, engage in a detailed discussion about proofs of concept (POCs) in the context of supply chain software and their limitations. Vermorel explains why Lokad frequently declines POC requests from prospective clients, citing potential harm to both the client’s company and Lokad.
Vermorel begins by acknowledging that POCs have been around for two decades and are not inherently detrimental. They function effectively when applied to a narrowly defined problem. The archetype of a situation where a POC would work well is choosing an email client like Microsoft Outlook or Gmail. This is a standardized problem with a known solution. The user has clear expectations and knows the pain points. This process is not transformative anymore; it merely involves transitioning from one email client to another.
However, the challenge arises when POCs are applied to supply chain optimization, which Vermorel refers to as “the opposite” of a narrowly defined problem. He suggests that it’s a transformative process for a company, characterizing it as a non-local problem spanning multiple locations and potentially numerous countries. The complexity is amplified due to the interconnected nature of the supply chain.
Further elaborating on the idea of an “open problem,” Vermorel describes supply chain optimization as the idea of wanting to optimize, which inherently requires measurement. The establishment of a meaningful measurement in monetary terms requires substantial effort, and the metrics obtained are only the beginning. The journey continues with refining the “numerical recipe” to assess the impact of actions on the company’s operations. He offers an example of assessing the precise cost of every stock-out or the specific impact on customer loyalty for every customer.
When asked whether POCs are ineffective exclusively for the quantitative supply chain or for the supply chain industry as a whole, Vermorel asserts that most supply chain problems are difficult to frame as POCs due to their interconnected nature. He explains that supply chains involve multiple parties - suppliers, clients, storage warehouses, and production plants. They present a distributed challenge, as opposed to a localized one, like optimizing a manufacturing process in isolation.
Vermorel underscores that supply chains are inherently connected to suppliers and clients, making it challenging to identify anything both local and meaningful. He cautions that while it’s easy to perform local optimizations in supply chains, these efforts typically result in merely shifting problems around. For instance, a significant effort to optimize one product at one location may inadvertently create issues elsewhere in the supply chain.
Vermorel explains that supply chain optimization is challenging for POCs because they tend to focus on micro-optimizations without considering the bigger picture. This can lead to problems for suppliers and clients. Additionally, lead times complicate the process further as it’s difficult to optimize a supply chain in a shorter timeframe than the lead time itself. This often results in POCs taking much longer than anticipated.
Chandler and Vermorel discuss the importance of looking at the bigger picture when trying to optimize supply chains. They mention that POCs often fail to consider the full lead times necessary for a true understanding of the supply chain. Vermorel advises adopting a capitalistic approach at every step of the methodology, focusing on what will not change regardless of the vendor or solution chosen.
One challenge faced during POCs is gathering and measuring data. Vermorel believes that you cannot optimize what you do not measure, so gathering data should be a priority. However, reality often makes gathering data for POCs more difficult than expected. This is due to the complexity of real-world situations, such as retail networks with multiple warehouses, production units, and distribution locations.
Vermorel provides an example of how historical demand can be difficult to assess accurately. Issues like stockouts, promotions, and other anomalies can skew historical order data, making it challenging to understand true demand patterns. When POCs encounter these problems, they often revert to classic forecasting methods, such as weekly time series forecasts. This approach simplifies the problem but ignores the complexities and nuances of supply chain optimization.
Backtesting, or using historical data to test forecasts, is another tool used in supply chain optimization. While it works from a statistical perspective, Vermorel argues that it only represents a fraction of the bigger picture in supply chain management. For instance, purchasing patterns can be affected by factors such as negotiated prices and minimum order quantities (MOQs), which are not accounted for in backtesting.
Vermorel highlights that supply chain optimization is not a one-dimensional problem that can be solved by focusing solely on forecasting accuracy. He argues that existing processes need to be revisited and adjusted to make optimization possible. The main issue with focusing on forecasting is that it overlooks the bigger picture and relies on a specific type of forecasting that may not be applicable to all situations.
Vermorel points out that supply chains have been digitalized for decades, but many companies still rely on Excel sheets and ignore the numbers generated by various systems. He suggests that companies should focus on the fundamentals, such as consolidating data into a data lake and creating meaningful documentation that reflects their supply chain’s semantics.
By focusing on the stable aspects of a supply chain, such as suppliers, production units, warehouses, and distribution channels, companies can craft metrics that reflect their long-term financial interests. Vermorel emphasizes that the main challenge lies in understanding the semantics of the data and ensuring that the right data is available for optimization.
When considering alternatives to POCs, Vermorel proposes focusing on what does not change in a supply chain and looking for a vendor that can provide a data-driven, holistic perspective. While POCs can offer some insights, he warns against expecting too much from small-scale experiments, as complex problems require more in-depth approaches.
He also suggests that working with vendors should not be a long-term commitment. Companies can engage in lean initiatives and assess the vendor’s performance on a weekly basis to ensure progress is made. If the progress is unsatisfactory, it is better to terminate the relationship early rather than continue with a suboptimal vendor.
Full Transcript
Kieran Chandler: Today, we’re going to understand exactly why POCs don’t work and also understand what are the alternatives that are open to a customer in order to choose between the multitude of supply chain software out there. So, Joannes, POCs have been around for two decades now, so they can’t all be bad. In which industries do they actually work?
Joannes Vermorel: Proofs of concept work well when you have a close, narrow problem that you want to address, when you don’t have to think of massive open possibilities. For example, the archetype of an area where a POC would work well would be if I ask you to choose your email client because you’ve used an email client, let’s say Microsoft Outlook or Gmail, for years. So, you know what you should expect, you know your pain points. This is a fairly standardized problem with standardized solutions, and you will make a very precise comparison because you know exactly what to look for. For your company, this is fundamentally something that is not transformative; it was transformative decades ago when people started to use email altogether, but at this point, it’s just to move from one email client to another. POCs work well for those super tactical, well-defined problems. The big challenge is that the quantitative supply chain is kind of the opposite. It’s something that will fundamentally transform your company, and it’s fundamentally something that is completely non-local. You can’t execute a supply chain locally from your desk. It’s because there is a network that spans over many locations, potentially many countries, that things get really complicated.
Kieran Chandler: So, what is it that characterizes that as an open problem?
Joannes Vermorel: Supply chain is first an idea that we want to optimize, and to optimize, you need to measure. Just the very fact that you want to measure things in euros or dollars, it takes a considerable amount of effort to obtain a measurement that actually makes sense. The metric, the starting point, is actually a journey to refine the way your numerical recipe assesses whether you are doing something good or bad for your company. Obviously, serving customers is good, having an enormous amount of stock-outs is bad, but the question becomes exactly how do we assess the precise cost of every single stock-out, the precise impact on customer loyalty for every single customer?
Kieran Chandler: So, when you say proofs of concept fundamentally don’t work, is it just for the quantitative supply chain that they don’t work, or is it for the supply chain industry as a whole?
Joannes Vermorel: Most of the supply chain problems are very hard to frame as proofs of concept. By definition, supply chains involve multiple parties: you have many suppliers, many clients, storage warehouses, production plants. It’s a very distributed challenge in essence, as opposed to something that would be extremely local, such as the optimization of a manufacturing process that is completely isolated from the rest of the world. Supply chains are completely connected to your suppliers and your clients. So, pretty much by definition, it’s hard to find anything that is both local and meaningful, because the problem with supply chains is that it’s very easy to perform a local optimization. The point is that when you do that, usually what you end up with is just moving problems around. So, yes, you’ve done a great amount.So, you’ve created a problem for your supplier and maybe for a couple of clients as well by doing that. You haven’t solved the big picture; you’ve just micro-optimized something locally. That makes supply chain very challenging for proofs of concept in general. And then there is another layer of difficulty in the case of Lokad. Because we want to tackle the future and the uncertainties, there are the lead times that have to be taken into account. So whenever we are dealing with an industry where lead times are, let’s say, you have to plan and execute things thinking three months ahead. It means that no matter how you optimize the numerical recipes for your supply chain, one iteration takes pretty much as long as your lead time to come to pass. So, that means that you can’t really do anything in a timeframe shorter than your lead time, and more realistically, you will need like two or three iterations. So, if you have lead times of about three months, and we are talking about two or three iterations, we are talking about six to nine months. It’s not abusively long for enterprise software, but we start to get very far from what people think of when they think of a quick proof of concept.
Kieran Chandler: Let’s kind of unpack that a little bit. So, what you’re saying is that from a proof of concept, it’s basically only looking at a small picture, and in order for a solution to really work and to prove itself, it needs to be looking at the whole bigger picture. And then the second thing you mentioned was to do with lead times, and you’re saying that a proof of concept is normally quite short, and so we’re not taking into account full lead times to really get the results and understand what’s happening. How long should you really persevere with a proof of concept before you’re actually going to see any results?
Joannes Vermorelr: The point is that if you wait until you finalize your measurements and it’s absolutely clear that you can measure the benefits and codify, it’s going to take too long. That’s why typically, what we suggest is to have a methodology that is highly capitalistic at every step and is driven by the vision. What I mean by capitalistic at every step is that no matter how you want to optimize your supply chain, you will need data to execute this optimization. The process is going to take a bit of time, but it is kind of independent of whichever vendor or solution you pick, even if you want to do it internally or with an external company. You will need to do this process. If you can execute your initiative in a way that you focus on what will not change with respect to whatever vendor is chosen, if any, at the end of the initiative, then you can be highly capitalistic. For example, you cannot optimize what you do not measure, so you should start gathering data for measurements. This very effort is probably going to go beyond what you think is eligible for a proof of concept-sized effort.
Kieran Chandler: That’s probably quite surprising to a lot of our viewers because, in proofs of concept, gathering the data should be the easy bit that’s supplied when you start the proof of concept. So why is it so difficult?
Joannes Vermorel: It’s always difficult because reality has its own ways to make sure that your proof of concept attempts fail in the most miserable ways. But more seriously, let’s pick a real situation of what is happening. Let’s say you have a retail network that involves a couple of warehouses, a couple of production units, and maybe a couple of distribution locations. Now you start your initiative where you say.
Kieran Chandler: Let’s talk about clients that are happy with how you serve them. So, you realize that you want to start your proof of concept, and you find that getting all the transactional data is more difficult than it seemed at first. Why is that?
Joannes Vermorel: Well, for example, you could say client orders should be easy. Yes and no, because you can have a lot of odd situations, such as a client sending you an order you cannot fulfill due to a stock out. You diligently record in the system that the order couldn’t be fulfilled. The next day, the client places another, even bigger order. Why? Because you didn’t fulfill the order from the previous day. But if you had fulfilled the first order, there probably wouldn’t have been a second order the next day. So, you want to reflect historical demand, not just the raw historical stream of orders from the client that were impacted for various reasons. You realize it’s more tricky than it seems.
Kieran Chandler: So what typically happens with a proof of concept initiative when you face all those problems?
Joannes Vermorel: You try to find ways to simplify the problems and inevitably circle back to a classic forecasting proof of concept, like a weekly time series forecast. You think in terms of forecasting demand based on historical demand, and you can back test that immediately. You ignore stock outs, promotions, and all the bizarre factors. The end result is something that fits the definition of a proof of concept, but by doing that, you’ve completely deviated from the problem you were trying to solve.
Kieran Chandler: Let’s talk a bit about back testing. That’s looking at historical data in the past, making forecasts, and comparing the two. Why doesn’t back testing really work?
Joannes Vermorel: Back testing works from a statistical perspective, but it has limits. From a supply chain perspective, back testing is just a tiny numerical tool that is only a fraction of the whole picture. If you start thinking about optimizing the purchasing patterns of your teams, it turns out that maybe all those minimum order quantities are not carved in stone. Maybe those infrequent purchases are caused by the fact that your purchasing team negotiated low prices, but with exceedingly high minimum order quantities. This forces your team to put extra delay between purchase orders, which complicates everything and inflates lead times.
What I’m saying is that performing optimization is not just a one-dimensional thing, such as decreasing error from a forecasting accuracy perspective. It’s also about embracing and revisiting all existing processes and adjusting what is being done when possible so that optimization actually becomes possible. Also, you need to think about what you’re actually measuring. The problem is that these small proof-of-concept projects, which inevitably circle back to time series accuracy benchmarks, end up having an error expressed as a percentage instead of dollars. Again, we are very far from any business fundamentals.
Kieran Chandler: So, the real problem with forecasting is that you’re just focusing on that aspect and not looking at the bigger picture? It’s a very specific type of forecasting. Gathering all the relevant data is difficult, so you can be assured that your proof of concept is going to be a classic time series forecast with weekly data in and weekly forecast out, and this is supposed to be it. I guess what we’re kind of describing today is coming from a bit of pain of experience, the things that you’ve tried in the past and POCs you’ve done. So, what are some of the key problems that you’ve experienced when you’ve done POCs?
Joannes Vermorel: The worst part is that sometimes you can completely succeed in the POC, and that was probably the thing that hurt the most. As an enterprise software company, you don’t win every single lead that comes your way, so losing leads is just a matter of life. What really hurts is when you win the POC because, yes, in the POC you perform best, potentially by a huge margin, and then when you try to translate that into a production situation, things completely blow up and do not meet any of the client’s expectations. You realize that it’s because you moved from a toy problem to a real-world problem. Maybe whatever you had done for the toy problem was working very nicely on this limited scope with a well-defined metric, but it completely ignored all the business fundamentals. When you transition to the production situation, you realize that your solution is not even close to providing an answer.
The worst part is that because supply chains have been digitalized for a couple of decades now, for large companies, you realize that the solution you’re bringing into the company is just going to be like the previous attempts. It will produce a mass of numbers that will be completely ignored by everyone. Inevitably, all the supply chain teams fall back to their usual Excel sheets, completely ignoring the dysfunctional numbers produced by yet another system.
Kieran Chandler: Let’s look at things now from the perspective of a company. The nice thing about POCs is you can see the different ways that software vendors approach a problem and how they’re actually reacting to certain challenges. Other than that, what are the alternatives that companies can use instead of a POC? What else could they do to see how different companies perform?
Joannes Vermorel: The alternative is to focus on the fundamentals. I was pointing out that there is the challenge of consolidating data. The modern way of doing that at present is to build a data lake. The step beyond that is to provide some meaningful documentation from a supply chain perspective. To be successful, you need to focus on what does not change. Your supply chain will probably not change in its most fundamental ways. You will still have suppliers, production units, warehouses, and distribution channels. There are plenty of things that are expected to be relatively stable, so focus on that. When you consolidate data, also consolidate the relevant documentation because the big challenge is always the semantics of the data.
What does this data mean? When we have an order date, was it the date when you produced the order to be sent to the supplier, the date when the entry was registered in your system, or the date when your supplier acknowledged receiving the order? There are a dozen possibilities. The question is, where are all those things documented? If you do not have access to the data and semantics of the data, there is no hope of ever optimizing anything. Again, for example, when I described a quantity supply chain approach, yes, there is Lokad as a cloud computing platform to just execute at scale all those numerical recipes. But the things that are highly capitalistic for the client are the consolidation of all the data that is relevant, understanding the semantics of the supply chain, crafting metrics that truly reflect the long-term financial interest of your company, which is difficult. Crafting the processes that play well with the physical constraints of your supply chain, and all of that is inside and largely independent from the toolkit that you use to execute those numerical calculations.
Kieran Chandler: If we start to bring things together and conclude today, PoCs have been around in the supply chain industry for decades now. I mean, can you really envisage a time when PoCs don’t exist at all?
Joannes Vermorel: Obviously, there will still be young people who don’t know any better who will ask for one more PoC. You know, that’s just a fact of life. And maybe they will be right, because again, there are situations, even supply chains, where a PoC is really the way to go. If you have a new barcode printer that seems to be compatible with the system, which has the old barcode printers, but the new one is just better, faster, cheaper, leaner, and whatnot. This is a problem where you can buy one more barcode printer and test it out in your favorite warehouse. So, there are situations and problems where a PoC is just the way to go.Yet, when you want to think of a supply chain-wide optimization challenge that is really taking a holistic perspective on your supply chain, I would say don’t expect too much from small-scale experiments. It’s just that inevitably, you will realize that the problem is complex, and if you do not try hard enough, you will just fail because you’ve not even tried hard enough at the problem.
Kieran Chandler: So, as a final key message from today, is it that PoCs can give you some sort of insight, but don’t expect too much, and they don’t really work as a whole?
Joannes Vermorel: Yes, and the opposite of a PoC should not be a ten-year commitment with a vendor. I mean, that’s also another thing. It’s not because you’re not doing a PoC that you can’t say, “I suppose we can work with a vendor. What we would like is to start initiatives, you know, a lean industry, so it doesn’t have to be very expensive, and then we will just move forward with you week after week.” Instead of paying attention to the fact that we are supposed to frame initiatives within ten weeks, it’s more like, does the speed of progress of the initiative satisfy you? From one week to the next, are you making progress at a satisfying rate?
If you pick a vendor and after three weeks you realize that things still appear to be very sluggish, then you should quit, even in a time frame that is shorter than a typical PoC. The question is more like, you think ten years ahead, but that’s your vision. You focus on an element that will be highly capitalistic for your company for the next decade. But when it comes to kicking an enterprise vendor out of the door because they are just not delivering, it’s more of a weekly assessment where you revisit if things are progressing and if there is some kind of momentum building up. You’re looking for all those small signs, with an indefinite planning in mind.
Kieran Chandler: We’re going to have to leave it there, but we’ll have to see if we’ve scared everyone off.
