00:00:06 Supply chain data processing challenges and Lokad’s terabyte scalability.
00:00:38 Improvements in Lokad’s processing capacity over the past year.
00:01:33 Explanation of Lokad’s platform and their scripting language Envision.
00:03:55 Comparison of Envision to other programming languages like Python.
00:06:27 Key insights and breakthroughs that led to Lokad’s improvements.
00:08:00 Tabular vs. columnar storage in databases and their efficiencies.
00:10:36 Challenges and costs associated with processing large amounts of data.
00:12:04 The impact of terabyte processing on supply chain scientists and work efficiency.
00:14:10 Real-world benefits of faster processing and handling larger datasets.
00:15:30 Importance of eliminating edge cases and the danger of long processes in quantitative supply chain initiatives.
00:17:43 Importance of an iterative approach to deal with real-world contingencies.
00:20:47 Challenges and implications of large-scale data processing in supply chain optimization.
00:21:10 Terabyte scalability’s impact on Lokad’s outlook and the future of supply chain optimization.
00:24:41 Closing thoughts and potential avenues for improving supply chain optimization.
Summary
Kieran Chandler interviews Joannes Vermorel, founder of Lokad, about supply chain optimization and handling vast data sets. Lokad has increased its processing capacity by 20 times since 2017, utilizing their domain-specific language, Envision, for large-scale data processing. Envision simplifies distributed computing in the supply chain context compared to generic programming languages. Lokad’s advancements have reduced data processing times from hours to minutes, allowing supply chain scientists to work more efficiently. Vermorel stresses the importance of agility and iterative optimization for successful supply chain initiatives. Lokad’s terabyte scalability enables a more holistic approach to optimization and plans to enhance expressiveness in Envision scripts to better address real-world supply chain situations.
Extended Summary
In this interview, Kieran Chandler, the host, discusses with Joannes Vermorel, the founder of Lokad, a software company that specializes in supply chain optimization. They talk about the challenges of handling vast amounts of data in the supply chain industry and Lokad’s developments in terabytes scalability.
Joannes mentions that 2018 has been a year of scalability for Lokad, with the company increasing its processing capacity by a factor of 20 compared to the previous year. This improvement has enabled Lokad to handle multi-terabyte data sets with relative ease, considering the current state of technology.
The company has achieved this by developing the Solocat platform, which is designed for quantitative supply chain optimization. The platform utilizes Lokad’s domain-specific scripting language called Envision. Envision is designed to address the specific needs of supply chain problems and facilitate large-scale data processing.
Before 2018, a single Envision script could only be executed on a single machine, even though the platform could hold several terabytes of data. The company had implemented some level of scalability by distributing machine learning calculations across multiple machines. However, tasks like sorting data were still limited to one large machine.
In 2018, Lokad rolled out a completely upgraded architecture for the backend execution logic of Envision scripts. The new architecture enables automatic parallelization across multiple CPUs and even entire fleets of machines. As a result, tasks such as sorting data can now be performed much faster by utilizing the processing power of dozens of machines.
When comparing Envision to generic programming languages like Python, Java, C++, and C#, Joannes explains that Envision is a domain-specific language tailored to handle supply chain problems. While generic programming languages offer the ability to do almost anything, they often require more complex configurations and a deeper understanding of specific libraries and frameworks to achieve distributed computing.
Envision, on the other hand, simplifies these technicalities, making it easier to handle distributed computing and process large-scale data in the context of supply chain optimization. This specialized approach allows Lokad to improve scalability and handle terabytes of data more effectively than using generic programming languages.
They discuss the company’s software, Envision, its benefits, and how recent advancements have improved supply chain analytics.
Envision is a scripting language specifically designed for supply chain problems, making it easier and more efficient to use than other general-purpose programming languages. It sacrifices some expressiveness for simplicity, but since it focuses on a specific type of problem, this trade-off is acceptable. Vermorel compares the ease of using Envision to writing advanced Excel sheets with fancy averages, sorts, and lookups.
One year ago, Lokad had already implemented a columnar approach to data storage for big data analytics, which is more suitable for batch analytics rather than real-time processing. This approach stores data by column, rather than by row, allowing for more efficient processing of operations that only touch a few columns. However, the drawback is that adding new rows requires working with every single column, making it less efficient for real-time processing.
When discussing the costs associated with processing large amounts of data, Vermorel emphasizes that the most expensive resource is the supply chain scientist (or data scientist), as they are scarce, require specific knowledge, and need extensive training. As a result, Lokad aims to optimize the use of these scientists’ time by reducing the time spent on processing data.
Previously, processing terabytes of data would take hours, but recent advancements have reduced that time to minutes. This allows supply chain scientists to stay focused on their tasks and move on to the next step more quickly, rather than waiting for results. However, it also means they have fewer opportunities for breaks.
The breakthrough of processing larger data sets more quickly has several real-world benefits. It allows companies to analyze and optimize their supply chains more efficiently and effectively, ultimately improving their overall operations.
They address the challenges of achieving optimization, the importance of agility, and the iterative approach necessary to handle real-world contingencies.
Vermorel identifies the primary danger in implementing a quantitative supply chain initiative as the time it takes to perfect the system, as it is crucial to handle all edge cases to avoid manual intervention. The longer it takes, the more likely the project will face disruptions, such as IT or ERP upgrades, or dramatic business transformations, which can render previous work obsolete.
To succeed, supply chain optimization initiatives need to be put into production quickly, which requires agility. Vermorel explains that while it might sound like a lot to accomplish in a few months, taking too long risks project failure. He emphasizes the iterative approach to optimization, which is necessary due to the unpredictable nature of supply chains and real-world contingencies that can derail numerical recipes.
Iterative optimization is essential for handling the endless stream of surprises that come with supply chain management. Vermorel explains that supply chains are complex, involving multiple markets, countries, suppliers, and product lines. It is impossible to brainstorm and list all potential technical challenges beforehand, so agility is crucial when fixing numerical recipes.
Kieran then asks Vermorel about the impact of terabyte scalability on Lokad and its future outlook. Vermorel states that the company has made tremendous improvements in scalability processing, which opens new opportunities for supply chain optimization. He clarifies that scalability is not just about dealing with larger companies, but also about achieving network-level optimization.
Vermorel highlights that true optimization requires considering the entire network as a single entity, rather than optimizing each store or supplier in isolation. Terabyte scalability allows Lokad to approach optimization more holistically, which in turn helps prevent problems from being displaced down the supply chain.
In the future, Lokad plans to use its gains in scalability to improve expressiveness, allowing for more fluent and direct ways to reflect complex real-world situations in their Envision scripts. This will make it easier to develop numerical recipes that effectively address real-world supply chain situations.
The interview concludes with Vermorel emphasizing the importance of agility, iterative optimization, and holistic approaches to supply chain management.
Full Transcript
Kieran Chandler: Today on Lokad TV, we’re going to understand the sheer scale of data that is managed within a supply chain and also discuss how Lokad has developed terabyte scalability in order to deal with that. So, Joannes, today we’re going to be talking a bit about the research and development done at Lokad over the last year. What have you been working on in 2018?
Joannes Vermorel: 2018 has been the scalability year for us. We have been working almost continuously for a year in increasing the scalability. Compared to the same date last January, we have increased our processing capacity by basically a factor of 20, which puts us well within the range of processing multi-terabyte data sets relatively at ease. There is no such thing as easy terabyte processing considering the technology as it stands today, but it can still be made relatively straightforward.
Kieran Chandler: A factor of 20 sounds like an incredibly large improvement. What sort of steps did you have to take to reach this improvement?
Joannes Vermorel: Lokad is a platform designed for supply chain optimization. Technically, it’s accessible through scripts written in our own homemade scripting language called Envision. This language is designed not only to specifically address the needs of supply chain optimization but also to facilitate large-scale processing. Until last year, a single account for one of our clients could hold several terabytes of data, but a single script or piece of code would be executed on a single machine. We had already implemented scale-out, so the idea that you can distribute the calculation over many machines for specific components such as machine learning. However, broadly speaking, when you were just sorting the data, it was happening on one large machine. In 2018, we rolled out a completely upgraded architecture for the backend execution logic for Envision scripts. Nowadays, you have automatic parallelization across not only multiple processors and CPUs but also a fleet of machines. This means that a single script that is performing a simple operation, such as sorting the data by date, product, store, or price, can happen over a fleet of dozens of machines, making it much faster.
Kieran Chandler: Let’s talk a little bit about Envision. How does Envision work in making these improvements and improving scalability compared to working with other programming languages like Python or something like that?
Joannes Vermorel: Envision is a domain-specific programming language, while Python, Java, C++, and C# are generic programming languages. With a generic programming language, you get the capability to do pretty much anything, but the price you have to pay is that there are classes of technicalities that surface into the programming environment. For example, you can do distributed computing with Python, but you need to write your code in a way that takes advantage of specific libraries and frameworks to have those calculations happen over a fleet of machines. Additionally, you will have to configure a cluster of machines yourself so that it can execute this distributed logic. If you have several concurrent programs that execute over the same cluster because you have different scripts or different users, it becomes more complex. Different needs… Well, we need to have all the plumbing in place to share the resources, you know, the bandwidth, the disks, the CPUs, and whatnot. And all of that is precisely what Envision does in a way that is completely built-in for you. So, what you lose is that you lose a lot of expressiveness, which is okay because, again, Envision can survive. It’s not broken, despite having lost a lot of expressiveness, because we are specialized in a specific type of problem, which is basically supply chain problems. So we don’t try to solve every single problem. We don’t try to write an engine to play chess or to do 3D modeling. It’s very geared towards specific types of problems.
Kieran Chandler: Okay, so what you’re saying is it’s a lot more simplistic to use Envision rather than another sort of programming language?
Joannes Vermorel: Exactly. I mean, writing a script that processes tables containing several billion lines to perform the sort of analytics that you would expect in a supply chain context, such as estimating the cost and impact of stockouts across a large network for the last three years, is something you could do with just literally a dozen lines of code. But code that is easy to write. When I say easy to write, I mean it’s never that easy to write code, but it’s no more difficult than, let’s say, writing an advanced Excel sheet that does fancy averages, fancy sorts, and fancy lookups over the data.
Kieran Chandler: Okay, and what were some of the key insights and breakthroughs that led to these improvements?
Joannes Vermorel: Just to understand where we were starting from, one year ago, Lokad had already many big data ingredients in place. One of them is to have a columnar approach to data storage. So, long story short, traditional SQL databases adopt a tabular storage. It means that when you have a table, you are storing lines of data, and if you have a line, it sits together, and that’s your unit. So basically, that makes tabular databases very efficient at performing small updates or read/write line by line.
In contrast, over the last decades, whenever you want to do big data analysis, what has been developed is a columnar approach. So basically, when you have a table, let’s say you have a table with 20 columns, you are going to store data packed by column. So, what does that mean? It means that when you perform an operation, such as for example, “I have the per-unit price, and I have the quantity. Let’s say you have a table that represents sales history. Now you want to know the total amount of sales, so what do you need to do? You need to multiply the price by the quantity and do the sum of all of that across the table.” Well, it turns out that your table might have 20 columns, but the operation that I just described is only touching two columns. So if you have a columnar storage, the main upside is that every operation that touches only a few columns, those few columns will be processed, and the rest can be just ignored. If you have a tabular format, well, the other columns are still completely in the middle, and you can’t really skip them.
But the drawback could be that if you’re adding additional lines, if you’re working in real-time, if you’re having a columnar approach, if you’re adding a line, you’d have to work with every single column.
Kieran Chandler: So, when you’re adding a line, you have to touch 20 columns, making it relatively inefficient. Columnar storage is more suitable for batch analytics, not exactly real-time, but more the sort of analytics you’re interested in for supply chain. It can be refreshed every minute, but not every millisecond. Let’s talk a little bit about the costs associated with processing power and working with large amounts of data. How has implementing terabyte scalability affected the costs of working with data?
Joannes Vermorel: Actually, quite a lot. The most expensive resources when you try to optimize a large-scale supply chain are the supply chain scientists. These people are not free, and it’s a challenge to find and train individuals with the right talent. The scarce resource is not CPUs, disks, or bandwidth, which can be cheaply built from your favorite cloud computing platform, but rather the talented people you have to work on the problem.
When you enter the realm of terabyte processing, everything is slow. Crunching a terabyte of data can take quite a while, especially if you’re doing non-trivial calculations involving probabilistic calculations with random variables. This could take hours, but now it’s taking minutes. For supply chain scientists, this means they can stay focused on the task, get the results, and move on to the next thing instead of being stuck waiting for the results. So, if anything, it’s bad news for our supply chain scientists because they get to have fewer coffee breaks.
Kieran Chandler: What does this breakthrough mean for the real world? Obviously, we’re able to work with much larger datasets and work with them a lot quicker, but are there any other benefits we’re seeing?
Joannes Vermorel: Yes, one of the key dangers, or root causes for failure of quantitative supply chain initiatives, is the time it takes to get it right, to get it working and completely polished so that you don’t have lingering edge cases. With the ability to process larger datasets faster, we can now address these issues more efficiently, which ultimately leads to better supply chain decisions.
Kieran Chandler: Your systems are good, but there’s still one person who remains purely manual. This requires a lot of manpower to manually go through all the edge cases that are not properly handled. This somewhat defeats the purpose of having advanced machine learning automation for optimization. The danger is that the process to eliminate all the edge cases, to really sharpen the decision-making numerical recipe, can take too long.
Joannes Vermorel: Yes, and that delay is lethal because at some point, people lose faith in the initiative’s success. If you’re slow, chances are that IT will be disrupted in the middle. For instance, if your project takes two years, there’s a one in three chance that there will be an ERP upgrade in the middle. You start to optimize supply chain, and then there’s a massive change within the company’s idea. Much of what you’ve done falls apart just because all systems are being changed.
The longer you wait, the more things can disrupt your project. The business itself can undergo dramatic transformation which makes your earlier work obsolete in light of the new business changes. There are a lot of driving forces that put pressure on getting the product into production quickly. If you can’t succeed within a few months, chances are that your initiative will fail. That’s where agility is absolutely required.
It might sound like a lot, a couple of months, but when you touch terabytes of data and it takes hours to get a result, you’re essentially tweaking your numerical recipes once or twice a day at worst. Things start to become very slow.
Kieran Chandler: So what you’re touching upon there is a kind of iterative approach to optimize supply chain. Why is there such an iterative approach? Why can’t you just press one button, use deep learning to learn and get a good forecast and good results?
Joannes Vermorel: Reality has a way of getting in your way. Supply chain is really about dealing with real-world contingencies. There are so many edge cases that just emerge from the very fabric of reality. For example, you start computing an ideal quantity to order, say 80 units. It’s the best quantity to order, but pallets only come in zero units or 100 units because they’re packed over a pallet. 80 is not an option, so what do you do?
There are dozens of situations like this. You might think the value of your inventory gradually decays over time, but it doesn’t. Why? Because there’s a shelf-life with a specific date. When you reach the shelf-life limit, your inventory value drops from something to zero or even negative because now you have to pay to dispose of this worthless stock.
The reality is that supply chains are complex. If we’re talking about companies that operate over many markets, multiple countries, with many suppliers, potentially dozens of product lines, there’s a lot of complications. It’s wishful thinking to believe that you can just sit down in a room and brainstorm with the team all the technical challenges that will emerge down the line with the initiatives. You cannot just say… Sit down with your supply chain team and say, let’s list all the small things that will derail the numerical recipe that generates the optimized supply chain decisions. Reality, as always, has a way to surprise you because you’re going to discover bizarre things like a certain country having a strange tax on something, for example. Circling back to your initial question on the iterative approach, the only way to cope with this endless stream of surprises is to be extremely agile in fixing your numerical recipe when you face something unexpected.
Kieran Chandler: Now let’s focus on terabyte scalability and start to bring things together. How does this change things for the outlook of Lokad, and what does it mean for the future?
Joannes Vermorel: We have made tremendous improvements in terms of scalability processing. This development opens a lot of changes that were previously inaccessible to us. It’s not just about dealing with larger companies, there are plenty of ways to cheat, for instance. Consider a large retail network. If your approach to optimizing a large-scale retail network is to process each market in isolation, and if you have, say, 200 markets, you would only need 200 machines, one per market, and it would scale. But this method will not yield the supply chain improvement you’re seeking because you’re not optimizing at a network level. You are locally optimizing each store in isolation, completely ignoring whatever is happening in the rest of the network.
Kieran Chandler: So these 200 machines wouldn’t be communicating with each other?
Joannes Vermorel: Exactly. If you have independent silos, it’s very easy to scale your processing. But the challenge is when you start thinking of your network as one entity where everything is connected. You can actually ship inventory and materials across the network in pretty much any way you want. Although there are plenty of ways that are not profitable and don’t make sense, it’s possible. And some of these subtle ways might be very good ideas and profitable ones. You need a system that can crunch everything at once, in a monolithic way. What does that mean for the future? It means that it opens tons of avenues for better, holistic optimization. The curse of supply chain optimization is that you don’t solve the problems; you just displace them. You may look at one area and optimize it, but all you’re doing is moving the problem down the line or back to the supplier. Another avenue is that now that we have significantly gained in scalability, we’ll probably try to go back to expressiveness. Basically, we aim to express complex situations that happen in the real world more directly and fluently in those Envision scripts. This way, it’s easier to deliver numerical recipes that are a good fit to address real-world situations.
Kieran Chandler: That sounds great. We’re going to have to wrap things up there, but thanks for your time today. That’s everything for this week. Thanks very much for watching, and we’ll see you again next time. Bye for now.
