Bob Trebilcock: Welcome to The Rebound where we'll explore the issues facing supply chain managers as our industry gets back up and running in a post-COVID world. This podcast is hosted by Abe Eshkenazi, CEO of the Association for Supply Chain Management, and Bob Trebilcock, editorial director of Supply Chain Management Review. Remember that Abe and Bob welcome your comments. Now to today's episode.

Welcome to today's episode of The Rebound, Behavioral Economics or Are Your Biases impacting your planning? I'm Bob Trebilcock.

Abe Eshkenazi: I'm Abe Eshkenazi.

Bob: Joining us today is Jonathan Karelse. Jonathan is the CEO of the consulting firm, NorthFind Management, and the author of Histories of the Future: Milestones in the Last Hundred Years of Business Forecasting. Jonathan, welcome.

Jonathan Karelse: Pleasure to be here.

Bob: Well, it's a pleasure to have you. There's an old saying that the one thing we know for sure about a forecast is that it's wrong, and we're not just talking about the weather. In Jonathan's book, he writes that for all the technological advances in the field, forecasting remains the business of guessing, and that forecasts will always be, to some extent, wrong.

Jonathan has some ideas on why forecasts are wrong, how our biases impact the accuracy of our forecasts and demand plans, something called behavioral economics, and how to improve them. Let's get started. Jonathan, first question. I've talked to a lot of supply chain managers over the last couple of years, and without question, I think planning was the first casualty going into COVID, or at least the first supply chain casualty. It seems as if coming out of COVID, we're still struggling to get it right. What happened?

Jonathan: Well, there's, as usual a couple of things or a few things actually. To begin with and going back to your opening comment, which is exactly on point, I think a lot of us have a misguided relationship with what forecasting is or more importantly, what forecasting should be. We got to the point somewhere along the journey of thinking that because we had all these technological advances and because we had decades of research, that somehow, we were at a level of sophistication that the future was ours to predict. The reality is no one can predict the future. We've gotten so comfortable with saying the forecast is always wrong, that we've forgotten what that actually means.

It literally means you're never going to get it all right. We'll talk more about that later, but there's some important lessons from that. Number one is I think we've spent a lot of time trying to forecast things that aren't forecastable to begin with. The second is, and this also arises from the idea that we've come so far with technology, everybody and their sister right now is selling a solution that says it has AI and ML integrated. I want to be crystal clear for all the proponents of machine learning out there that I think there is an application for it, but I think there's a lot more development in practice that has to actually happen.

COVID is a great example of why, for the very long foreseeable future, we are not going to be, or shouldn't be, entirely dependent on statistically driven forecasts or machine-driven forecasts. When the landscape changes as dramatically as it did during COVID, you're going to need the benefit of the responsiveness and judgements, and business intelligence inputs that you can only get from humans. I said there were a few, that's two. If I were to put an additional layer on top of what else went wrong, it's that a lot of organizations have this idea that once they've got the capital T, capital P process, the forecast process in place, that they need to stoically stick with that.

The reality is as business conditions change, as the landscape changes, as the market shifts as it did during COVID, we have to be ready to abandon something or at least be open to modifying something that may have worked very well for the last 10 years when things are more steady state. To recognize that forecasting is just one of a bevy of tools available for operations management and operations strategy, we can never be too married to the idea that the forecast needs to be in any condition, a thing that we go through the process of creating.

Abe: Jonathan, really interesting that you're bringing up the concept of planning and this idea of behavioral economics. It sounds rather complicated, bringing in a behavioral activity with an economic or a quantitative activity. Give me a sense of how these relate to planning.

Jonathan: It sounds complicated and actually can be quite complicated. I think at the highest level, the reason we're talking about behavioral economics in planning is because every one of the elements of planning that has human judgment involved, and whether or not you're integrating business judgment or human judgments, there are still human judgments taking place.

For instance, even in a fully automated, statistically driven forecast environment, there were humans who were involved in the selection of the data and the business rules around treating outliers. There were humans involved in the selection of algorithms, and there are ultimately, obviously, humans involved in the decision of what to do with the forecast once it's been created, whether or not it was made by AI and ML or by some other means. Because of our human judgements all over planning, we need to understand to what extent judgements make us better or worse.

Now, the traditional way that we've understood human judgment is that it's the neoclassical economic view, which is that if you give people clear cut choices of varying values or varying utility, they're going to rationally choose the one that has the greatest utility to them. It intuitively seems correct and planning historically, and indeed all economics historically, was based on the premise that humans are rational actors. That so long as we've got a clear line of sight to the difference in the value between the options we're being given, we're going to choose the one that makes the most sense for us from a utility or value standpoint.

The reason we're talking about behavioral economics today is because the Nobel Prize winner, Daniel Kahneman and his collaborator, Amos Tversky, back in the early '70s, began making some revolutionary observations in psychological work they were doing. Basically, the spoiler is if you don't want to read all their work, humans aren't rational, at least not from a neoclassical sense.

Humans do things that are deeply impacted by unconscious drivers and biases. As much as we think we can be objective, as much as we think we can have an unbiased interpretation of the data that we're being given to forecast or interpret, we are impacted all the time by a host of unconscious bias and heuristic processes that move us away from what the neoclassical predicted outcome would be in those decisions. I'm not sure, Abe, if I've given you a less complicated answer to the question. Suffice to say, the reason we're looking at behavioral economics is to better understand people's relationship with decisions and in so doing, help them improve forecasting.

Bob: Jonathan, in your first answer, you were talking about, as much as we may want to try to get to a fully automated planning that we're not going to get away from human judgment. Part of what you were just talking about sounds an awful lot, there's similar work going on and how, from a financial standpoint, how we invest and how we make investment decisions. The biases that we bring to the table when we're making investment decisions, like, we're going to win, even though everybody else around us is losing, like gamblers,

Jonathan: Exactly, like gamblers.

Bob: Exactly, so talk a little bit about our biases and relate it to forecasting and planning.

Jonathan: Sure. again, that's a really big question, and I'm going to try to condense it, but it exactly comes down to a gambling type of behavior. To begin with, one of the first things that Kahneman and Tversky, and then later another Nobel laureate, Richard Taylor, found when they started looking at the way people's decisions in practice differ from how they're supposed to look. If they did exactly what neoclassical economics predicted they would, is we've got, rather than a linear relationship between risk and reward, it's both asymmetric and non-linear.

In other words, to make it simplest, it hurts a lot more to lose a hundred dollars than the benefit we feel from gaining a hundred dollars. That observation alone really drives to a lot of the, quote, "irrational behaviors" that we see particularly in investing behavior and in forecasting. Robert Goodman, not Robert Goodman, Paul Goodwin, sorry Dr. Goodwin, spent four decades researching, forecasting and practice and found that in greater than 80% of cases where humans make a change to a statistically driven forecast, they take away value.

In our own research, working with large organizations and their planning teams, we've found that humans are about four times as likely to make positive changes to forecast than negative ones. Why is that? Again, it goes back to this asymmetric and nonlinear relationship between risk and reward. We love the idea of materializing reward and prefer to gamble with the idea of risks. If I don't have to materialize it in my forecast, if there's a chance that it might not happen, I'm not going to put it there. That's just one example.

A couple of other maybe even higher or overarching types of bias that Tversky and Kahneman looked at were heuristic processes or mental shortcuts. One called the representativeness heuristic. This is, it's a heuristic that serves us well evolutionarily. If I've seen something, or if I've learned something at some point in my existence, rather than stopping to consider every single decision that I'm faced with during a day, my brain automatically tries to associate what I'm seeing with something that I've seen before and gives me a reflexive response to it.

A great example of this is, when I was a Neanderthal, and a tiger jumped out of the jungle towards me, and I jumped back to avoid him, that was a good decision. I lived to fight and eat another day, and that's a behavior I want to emulate. Many generations later, when I see a bus coming out of my peripheral vision towards me, I don't stop and consider its trajectory, its speed, and work through a number of different options. I just instinctively step back.

This heuristic process has a benefit, but the problem with the representativeness heuristic is that it is a shortcut. A lot of times in forecasting, rather than deeply querying the data, we eyeball data even when we think we're looking at it judgmentally. We can, for instance, see patterns or think we see patterns that don't exist. This is a manifestation of representativeness that's called the cluster illusion effect. We see this in more than 75% of demand planners that we work with.

False seasonality is another great example of representativeness where people are looking at data, they think they're seeing seasonality in it, they're selecting algorithms that include seasonality when in fact it's stochastic variability. Another great example is the availability heuristic. This is the heuristic that the shortcut that works by assuming that the information that sprung to mind very quickly is probably the correct answer. Again, this saves us from having to query our data banks every time we negotiate, which some researchers believe are between 30,000 and 35,000 decisions every day. You'd get paralyzed if you had to think about every single one.

Again, there's a benefit, but the problem again is, just because it's sprung to mind and just because you've heard the message a number of times doesn't actually make it true. In demand planning in particular, when an organization is beating a particular drum on quarterly results or a new account, subconsciously, this can begin to drive biases into the way that we should be interacting with otherwise sarcastic data so we begin to see trends or try to materialize trends that don't actually exist.

Abe: Jonathan, you're bringing up some really interesting concepts in terms of the human behavior and its connection to technology and trying to embed that within the technology, and you're indicating that it really is problematic. Bob and I were just at the Gartner Conference, and I can tell you a quick walk through the exhibit floor, and there are as many companies and organizations touting AI and machine learning as a way to enable this probalistic planning. It's much faster, it'll reduce the biases. Why won't technology take care of the challenges that you're identifying here?

Jonathan: Oh, technology, absolutely will. There's absolutely a time and a place for technology. I'm going to parse your question because the second part of it, which is why won't technology take care of it! I think is a great question and I'll answer that second. The first part is, all of the companies out there touting AI in their solutions. For sure, when you look at this year's magic quadrant, everybody that fared well is talking about how their solution is super-powered with AI.

Now, I have the benefit of just having written a book on the milestones of forecasting, and one of the chapters is on AI and ML. I can tell you from the research I did that if anyone can point to an example of "AI that's being used in software" that isn't part of the original discussion of AI that took place at Dartmouth College in the 1950s, I will be amazed. What I'm saying is there's nothing actually new. A lot of what is being called AI in these solutions is nothing more than the best-fit heuristic process that has existed in most forecast engines since the 1980s.

Most of what people are calling AI is not actually AI. There are some bona fide ML-driven solutions beginning to pop up. If you look at the last couple of iterations of the M Competitions, then certainly you can see machine learning has matured a long way compared to the stuff that was being touted 15, 20 years ago. All of it is still ultimately predicated on the belief that if I can get a computer with AI or ML or statistical methods to interpret the patterns that exist in the history of my data, and I can predict them into the future, I'm going to get a good forecast. When you do that, you don't have any human bias. That's true. When you do that, you mitigate the downside of human judgment, no question.

The problem is, as many companies found by about March or April of 2020, when the landscape shifts substantially, ML-driven models and statistically driven models can no longer interpret history as a predictor of the future because they aren't. The future no longer looks like what the last 1, 2, 3, 5, or 10 years looks like because there's been a seismic shift in consumer behavior. Organizations that try to put all of their eggs into the AI and ML basket find themselves exposed in moments where there are substantial changes in demand and where the patterns of the past can no longer predict the future.

In many, I would say even in most cases, statistically driven forecasts, whether or not they have, what various people's marketing departments would call AI, are going to do a great job when you've got steady-state products. When things begin to change, a computer can't possibly understand what that means, and it will take judgment. Having a process in place that allows you to flex judgmental muscles as well is a way to augment the benefit of that unbiased statistical baseline.

Bob: The example you used when conditions changed it's a great segue for the next question. We've just come through and are going through an extraordinary time where it's almost like coming out of COVID has been as unpredictable as going into COVID. Everybody got whacked, but not everybody got whacked as bad as some others. Two parts, can you think of some companies whose forecasting plans were better than their peers getting through COVID or getting through this time? If so, what do you think they did differently? What can we learn from the folks that managed it?

Jonathan: I don't believe anyone did a good job of forecasting during COVID because no one had the basis for doing so. I think, A, we're looking at shades of bad, and B, anyone that outperformed the group, at least early on, got lucky. Before anyone screams heresy, I'll tell you, I had Spyros Makridakis, who's the father of the M Competitions, one of the most cited Google scholars on the topic of forecasting on my podcast a couple of months ago. His number one ingredient in the recipe of success in forecasting is luck.

That's the reality. No one knew what COVID was going to do. No one could have predicted what the impact to consumer buying behaviors was. I don't think anyone whose forecast errors were worse than their peers can say, "Oh, we were better at forecasting". I can point to companies that fared much better though. This goes back to my earlier comments, which is that we shouldn't be calibrating our operational strategies, or indeed our corporate or financial strategies, on the expectation of a great forecast, because that rarely happens. We can't predict the future. We can just get directional insights, and some of us can do it better than others, but that can't be the entire basis of our strategy.

The companies that fared well during COVID were companies like HEB in Texas, who had already built a robust risk management and business continuity planning framework, already knew, "When something happens to disrupt our raw material supply or our finished good suppliers or customer demand, this is how we will react." They were able to respond much more quickly because they didn't have to think on the fly. They had to adapt and they had to tweak, but they already had a well-articulated risk management plan and business continuity planning plan in place.

This, to me, is the major takeaway for organizations during COVID. Do the best you can with your forecast, but understand that the forecast is one in the overall toolbox of ways that you both delight your customer and delight your shareholders. It's one of the various things you can do. To prepare for the next major supply chain disruption, it's not about somehow getting way better at forecasting it because the entire definition of a Black Swan event is that it can't be forecasted. It's do the hard work of actually articulating and testing a business continuity plan in advance so when you need it, it's ready.

Interestingly, I believe it was EY who did one of the many interesting studies they do in late 2019. They were proudly saying on LinkedIn and other places that something like 90% or north of 80% of senior executives they talked to were actively working on business continuity planning and had made it a top priority in their organizations. Cue COVID two months later, and obviously, very few of them actually were. To me, Abe, this is the number one thing we can do. You're not going to be able to predict the next catastrophe but what you can do is prepare for it.

Abe: Really interesting Jonathan a study was done a number of years ago about competencies and capabilities of individuals entering the workforce and what organizations were looking for versus what they found. Interestingly, where we were overweight wasn't technology, that it was much more powerful than we're actually using it. Where we're underweighted was in critical thinking and real-world experience.

You put a combination of a significantly sophisticated technology solution with individuals who really don't understand the data or the decisions that created the machine learning or the AI and you have the very bad combination. How do you safeguard against these biases with some of the gaps that we have within the talent as well as how powerful the technology is today? How do we marry the two so that we are getting the best out of human judgment and leveraging the technology?

Jonathan: You’re good at big questions, Abe. That's another multipart one. In the 1980s when organizations began moving, certainly, they were still on mainframes but they started moving towards peripherals that could drive or could run forecast software. Forecasting technology moved out of the domain of hardcore programmers and statisticians and more into the, I won't say mainstream. It wasn't mainstream in the '80s, but it was getting there. You didn't have to be a hardcore programmer anymore because there were programs built for it. You didn't have to be as hardcore a statistician because a lot of the coding was baked in.

By the time you get through the late '90s and early 2000s, forecast software was a huge, if not multi-billion then at least approaching, billion-dollar industry. What they did very well was convince enterprises that you didn't have to find people that were really good at stats or really good at data science or really good at interpreting these patterns. You just had to get a really good piece of software and it would do all of that for you. To some extent, that's correct.

The problem is it's akin to how sophisticated the software is on a 787. Probably 85% of the time I could fly it because the software can do everything that a human is supposed to. The problem is it's in those 15% cases where I need to know what I'm doing that the gap between human capability and software gets exposed. The number one or the first thing I'd say to your question is we have to stop believing that software is going to do it all for us. There isn't a shortcut. You need people in demand planning, and indeed all global planning, that deeply understand, not only their role but also the assumptions and the mechanics and the math that goes into the tools they're using.

There's multiple papers that not a lot of people want to talk about that have looked at the coding in a lot of the very popular forecast engines out there and found basic issues with the math like the Chi-squared calculation. Now without getting too esoteric, Chi-squared is really important because this looks at the residuals. If your algorithms are not treating the errors properly, they're not priming the algorithm properly for the forecast. The entire forecasts are based on a misunderstanding of how well the model fits the data. I'm talking about very well-known pieces of software. If you have people that don't understand any of the math or don't understand demand planning mechanics, that never occurs to them, they never notice that.

The second piece of your question around bias is again understanding what the strengths and weaknesses are of both the statistical approaches and the judgmental approaches. There's a ton of drawback to judgment. Like I said, 80% of the time when we want to change a forecast, we make it worse. We have to be very clear about what the rules of engagement are. We have to, if we're going to interact with the forecast, if we're going to add judgment to the process because we think for some reason the technology hasn't kept pace with the changing market conditions or the latest information that we have, the scale of our intervention needs to be meaningful there.

There's no sense making minute changes when your error margins are 40% anyway. It needs to be financially impactful. Tweaking the forecast on parts of your portfolio that generate no income anyway is literally a waste of money. The goal in forecasting is not to make a perfect forecast, it's to make more money for your company. You want to focus your efforts where they matter.

The third thing is when you're deciding whether or not to make a change, you have to know something definitively different about the future than the past. Otherwise, you're trying to beat a computer at what it's best at, which is finding patterns in the past and projecting them into the future. Once you've passed those three criteria, you're now in a place where if the business intelligence you have is good, if the judgment is good, you have a higher probability of being able to add value to the forecast.

That's why fingers should be kept off of the forecast except in this specific circumstance. When you marry the two together, Dr. Goodwin's research shows we can consistently outperform an either exclusively judgmental or exclusively stats-driven process over the long run.

Abe: Really interesting. I think this supports research that was just done with our risk committee where it identified the two top issues for organization trends going and that its advanced automation and analytics was number one and the second was stacking and mirroring those two. I think you're highlighting the challenges in marrying those two to get an effective planning for an organization. If you're recommending to or you're advising an organization do you start with the automation or you start with the people first?

Jonathan: I appreciate the softball after all the big questions. Obviously, you start with the people. Despite how well a lot of software companies have done at marketing the idea that, "Don't worry about master data management, don't worry about process. The implementation of this software will drive all of that for you," it's utter nonsense. You need to have a process that's robust, and you need to have people in the seats that know what they're doing and are well-suited for the organization and the tasks they're doing. Once you've got that, you can begin looking at automating the non-value-added parts of the activity they do, but starting that way is a recipe for disaster.

Abe: Jonathan, I can't thank you enough. That is all the time that we have today. Special thanks to our guest Jonathan for providing a wealth of information and I think a topic that will continue to be on everybody's radar. Finally, a special thanks to all of you for joining us for this episode of The Rebound. We hope you'll be back for our next episode. For The Rebound, I'm Abe Eshkenazi.

Bob: I'm Bob Trebilcock.

Abe: All the best everyone. Thank you.

Bob: The Rebound is a joint production of the Association for Supply Chain Management and Supply Chain Management review. For more information be sure to visit asc.org and scmr.com. We hope you'll join us again.