![\"\"](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/04\/How-To-Overcome-The-Confidence-Killer-That-Destroys-Most-Predictive-AI-Projects-.jpg\")
<\/p>\n<\/p>\n
Originally published in\u00a0Forbes<\/a><\/p>\n When Henry Castellanos first presented his machine learning model to his company\u2019s executives, he found himself fighting off a certain self-doubt that is so common among data professionals, it\u2019s almost universal.<\/p>\n On one hand, his model looked great. It did a sturdy job predicting which dental patients would fail to show for an appointment, so that a medium-large chain of dental offices could strategically double-book high-risk time slots \u2013 much the same as airlines overbook flights. The project promised healthy returns. If Henry\u2019s model predicted well enough, the business could dramatically reduce the high cost of empty dental chairs, while largely avoiding the repercussions that result when two patients show up for the same appointment.<\/p>\n But on the other hand, Henry\u2019s model sucked \u2013 in comparison to a magic crystal ball. It was about two times better than random guessing, but a magic crystal ball would have well outpredict it by flagging no-shows and only no-shows, without error. To be specific, if you used Henry\u2019s model to flag the top 10% most risky patients, those most likely to be no-shows, then about half would turn out to actually fail to show. That is about two times better than random guessing (since about one quarter of all appointments were no-shows).<\/p>\n It didn\u2019t necessarily bother Henry that a hypothetical clairvoyant model would defeat his. Like pretty much all credentialed data scientists, he knew that crummy models are valuable<\/a> \u2013 that magic crystal balls are only a fantasy and the best you can usually hope for from ML is predicting better than guessing. And yet, predicting better than guessing is generally more than sufficient to improve an operational \u201cnumbers game,\u201d delivering a strong bottom-line win<\/a>.<\/p>\n But Henry still had to sell the business on actively using his model.<\/p>\n The Machine Learning Industry\u2019s Routine Failures<\/p>\n As the meeting began, Henry could just feel it: The numbers were sound, but they weren\u2019t going to convince the executives<\/a>.<\/p>\n \u201cUltimately, I didn\u2019t really feel confident,\u201d he told me during a video interview<\/a>. \u201cI didn\u2019t have a direct answer to the question of whether or not my model would be actually valuable. I wondered how I could really communicate what using the model would mean operationally and financially.\u201d<\/p>\n Henry had technically validated his model, which is an industry norm that\u2019s generally \u2013 but wrongly \u2013 considered sufficient. The internal tug-of-war that he then experienced is endemic to the predictive AI<\/a>profession. In this field of technology, we\u2019re taught to make sound models, but then to screen them only with regard to their relativepredictive performance \u2013 that is, their technical performance \u2013 without any data-driven estimation of the absolute business value they would deliver if used.<\/p>\n This standard but flawed practice fails to heed an obvious universal maxim: You can\u2019t sell something without first understanding the customer\u2019s problem and seeing things from their perspective. When it comes to predictive AI projects, what\u2019s being sold is the use of a model. And a data scientist\u2019s business-side counterpart just doesn\u2019t care that a model predicts \u201ctwo times better than guessing.\u201d<\/p>\n Instead, they care about money \u2013 or other KPIs.<\/p>\n Q: Is Your Model Good? A: Who Knows?<\/p>\n To be clear, as a nerd myself, I do care indeed about that kind of technical measure, which confirms relatively good performance. It means that the model works as it was trained to do. ML has found patterns that hold in general \u2013 now encoded as a model that can be employed to tip the odds in the numbers game known as \u201cdoing business.\u201d Predicting two times better than guessing means the model has a lift of two. Lift is one of a handful of standard metrics the ML industry uses to evaluate models. Other metrics include precision, recall, F-score and AUC.<\/p>\n But any of these highfalutin metrics alone fail to serve the customer and fail to serve the business. They all accomplish a variation on the same theme: They tell you that a model performs relatively well \u2013 yet reveal almost nothing about its potential absolute value. They\u2019re helpful, but not sufficient.<\/p>\n And so, by sticking to these standard metrics, data scientists usually fail to answer the most obvious question about the model they\u2019re trying to sell: \u201cHow good is it?<\/a>\u201d Without pinning model performance to value<\/a>, the answer to this question of model goodness remains subjective. Without an estimation of business value, you could just as easily argue that a model is \u201cbad\u201d as it is \u201cgood.\u201d<\/p>\n What an irony: The most formal, technical metrics leave things fuzzy. Without further insight, they leave the buyer\u2019s decision to the mercy of whim and whimsy. Usually, reason prevails and the underinformed decision maker scrubs the launch. The model is never used and the project realizes no value.<\/p>\n This dire mishap persists and persists. After decades of advancements and numerous waves of hype, predictive AI is still stuck, routinely following a process doomed to failure:<\/p>\n Train a model using the \u201crocket science\u201d known as ML algorithms (good!). Instead, Tell Stakeholders The Monetary Performance<\/p>\n After this typical data scientist experience of a nagging feeling that their sales pitch is missing something \u2013 and an initially dazzled but ultimately lukewarm reception from the stakeholder (aka customer) \u2013 Henry made a decisive, fundamental shift. He moved to showing his executive something that matters: profit. The model was projected to create an additional $500k<\/a> in annual revenue.<\/p>\n Henry provided visibility into exactly what the model was expected to do. By double-booking the appointments that it flagged, the business would avoid a certain number of empty dental chairs, saving hundreds of dollars on each occasion. This process would also sometimes wrongly double-book, each time causing inconvenience as well as some monetary loss (such as losing a dissatisfied patient) \u2013 but the bottom-line payoff looked great.<\/p>\n Walking into this meeting was an entirely different experience. Henry felt the self-assurance of a professional armed with the basis to land a sale. \u201cI felt like this provided the validation that I needed to have the confidence to go into meetings and say, \u2018This model would make money.\u2019\u201d Henry\u2019s boss \u2013 and his boss\u2019s boss \u2013 were psyched.<\/p>\n The lesson is clear: Data scientists, that nagging feeling, a certain lack of confidence, is telling you something. The solution to a business problem isn\u2019t just to predict relativelywell. The solution is to predict well enough to demonstrate it\u2019s absolutely valuable. When data professionals take the old but prevalent route and don\u2019t provide visibility into the potential value<\/a>, they\u2019re very unlikely to sell the business on using their ML model<\/a>.<\/p>\n For more detail about Henry\u2019s project, view this video webinar, demo and interview<\/a>.<\/p>\n \u00a0<\/p>\n About the author
\nEvaluate the model only in terms of technical metrics that fail to assess its potential value (bad!).
\nFail to convince business stakeholders to use the model \u2013 so most ML models fail to deploy<\/a>.<\/p>\n
Eric Siegel is a leading consultant and former Columbia University professor who helps companies deploy machine learning. He is the founder of the long-running\u00a0Machine Learning Week<\/a> conference series, the instructor of the acclaimed online course \u201cMachine Learning Leadership and Practice \u2013 End-to-End Mastery<\/a>,\u201d executive editor of\u00a0The Machine Learning Times<\/a>\u00a0and a\u00a0frequent keynote speaker.<\/a>\u00a0He wrote the bestselling\u00a0Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die<\/a>, which has been used in courses at hundreds of universities, as well as\u00a0The AI Playbook: Mastering the Rare Art of Machine Learning Deployment<\/a>. Eric\u2019s interdisciplinary work bridges the stubborn technology\/business gap. At Columbia, he won the Distinguished Faculty award when teaching the graduate computer science courses in ML and AI. Later, he served as a business school professor at UVA Darden. Eric also publishes\u00a0op-eds on analytics and social justice<\/a>. You can follow him on\u00a0LinkedIn<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"Originally published in\u00a0Forbes When Henry Castellanos first presented his machine learning model to his company\u2019s executives, he found…\n","protected":false},"author":2,"featured_media":13894,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[24,8362,25,1085,3328,6098,10726,10725],"class_list":{"0":"post-13893","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-ai","8":"tag-ai","9":"tag-analytics","10":"tag-artificial-intelligence","11":"tag-data-science","12":"tag-data-mining","13":"tag-predictive-analytics","14":"tag-predictive-analytics-jobs","15":"tag-predictive-analytics-news"},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/13893","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=13893"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/13893\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/13894"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=13893"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=13893"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=13893"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}