Two famous and oft-repeated aphorisms are:
All models are wrong, and some are useful
And:
Prediction is very difficult, especially about the future
The first one is attributed to renowned statistician George Box in a 1976 paper Science and Statistics, published in the Journal of the American Statistical Association. It has been repeated endlessly. That is also why they’re called models. No statistical or AI model can provide 100% certainty.
The physicist Niels Bohr is credited with the second. Bohr usually attributed it to Robert Storm Petersen Danish, an artist, and writer. The original author remains unknown (although Mark Twain is often suggested). I prefer Yogi Berra, “It’s tough to make predictions, especially about the future,” because it’s safe to assume Yogi never read Bohr or Twain.
Now that we’ve established all models are wrong, and prediction is difficult, how does that tie to model drift and Coronavirus? Let’s start with Evolutionary Biology. Prevailing theories of evolution before 1992 were various takes on “gradualism,” a term that speaks for itself. “Punctuated Equilibria” put forth in 1972 by Steven Jay Gould and Niles Eldredge advanced the theory that the vast majority of species originate in geological moments (punctuations) and persist in stasis.
We are witnessing a punctuated equilibrium today – just without the equilibrium. Models understanding and predicting supply chains, customer propensity, interest rate arbitrage are among thousands of analytical models that fundamentally remain consistent over time, and are affected by gradualism.
An example of gradualism? The move from retail stores to online purchasing did not happen overnight. I remember buying books for the first time from Amazon in 1996, twenty-four years ago. I had a smartphone almost twenty years ago, and broadband internet access for at least fifteen years. Internet business, smartphones, and broadband changed our economies and lives, but it took a decade or two. Coronavirus upended everything in two months. That is punctuation.
This is the first punctuated event where ML models were in wide use
Coronavirus is not the only punctuation in modern times. Other U.S.-based examples: the stock market crash of 1929, The attack on Pearl Harbor, the 2007-2008 financial crisis, and the subsequent recession. The difference is that none of these events occurred with the widespread use of machine learning and AI models in extensive use (there was limited use in 2008). Before ML, predictive models were based on small samples of small sets of data (relatively speaking). Today, ML models are driven by massive amounts of historical data to discern patterns from which to make conclusions.
Much of weather prediction is based on sensors on planes. With air traffic down 80 percent, how reliable are the existing models?
Chocolate candy bar manufacturing has long been used as a model to optimize production, based on a mix of raw materials and predicted demand (which is stable except for certain holidays, for example). But coca nuts are roasted and then pressed to make liqueur, cocoa butter, and cocoa powder. The proportions change every day based on market expectations and taste points of the principal ingredient, the cocoa-nut, which comes in many varieties and availabilities. But as Bloomberg noted, demand for chocolate has dropped significantly across the globe, a development pre-COVID models did not see coming.
Another example is an agricultural chemical manufacturer that modeled its fulfillment chain from factory to distributor to retailer to the grower – to increase volume and revenue by creating programs to educate the chain about its application. The model was based on data collected from 2015 – 2018. When it was time to roll out in 2020, a few factors changed abruptly:
- New tariffs imposed added pressure on growers.
- “Smart” implements coming to market spreading chemicals more efficiently
- Coronavirus closures cut off markets for farmers’ products, with 40 million unemployed.
- Climate change effects in 2020 are markedly different than the ML training period
None of these factors, which are significant drivers, were in play in the data used to train the models. By June 2020, Coronavirus had taken the lives of 110,000 in the US, with 6.66 million cases worldwide. And that all happened in three months or so, with no vaccine or therapeutics in sight. Under these circumstances, what should they do? Refresh the model, revise it, or scrap it? That’s the dilemma facing data science teams today.
And that’s not all. To effectively address this, we must also take on the problem of adequate data sets, a problem I took up in COVID-19 pandemic models – are Machine Learning models useful? As I wrote:
Machine Learning is best served with mountains of data, because its purpose is to find patterns in the data that are predictive. The availability of large datasets today plays right into this appetite. Still, the question is, can ML and AI techniques be effective with much smaller sets of data?
