{"id":41046,"date":"2025-09-03T14:53:10","date_gmt":"2025-09-03T14:53:10","guid":{"rendered":"https:\/\/www.europesays.com\/ie\/41046\/"},"modified":"2025-09-03T14:53:10","modified_gmt":"2025-09-03T14:53:10","slug":"how-leading-enterprises-really-measure-gen-ai-roi","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/ie\/41046\/","title":{"rendered":"How Leading Enterprises Really Measure Gen AI ROI"},"content":{"rendered":"

Everything you always wanted to know about how to measure the return on investment of generative AI (and agentic AI) comes down to one anecdote: The Soviet Nail Factory<\/a>.<\/p>\n

In the 1920\u2019s, Joseph Stalin had a vision for turning Soviet Russia into an industrial powerhouse capable of going head-to-head with the West. He introduced a series of Five Year Plans<\/a> designed and administered by Gosplan, the USSR\u2019s central planning agency. Gosplan set quotas for factory output. And they really weren\u2019t suggestions. If factory managers and their workforces missed those targets, losing their jobs was the least of their worries. Meeting them meant big bonuses.<\/p>\n

\"\"<\/p>\n

Managers did what managers always do. They managed to the metrics.<\/p>\n

That brings me to nails.<\/p>\n

The USSR supposedly had a shortage, so Gosplan mandated higher output. When quotas were set by volume, factories churned out nails so small and flimsy they couldn\u2019t hold up a picture frame. When quotas switched to weight, workers produced nails so massive that carpenters couldn\u2019t lift them, much less use them.<\/p>\n

On paper, production soared, and workers got big paydays. In reality, no one had a usable nail.<\/p>\n

There is healthy debate about whether the Soviet Nail Factory story is pure urban folklore or an exaggerated story based on historical facts. That didn\u2019t stop economists, years later, from using it to amplify what is known as Goodhart\u2019s Law.<\/a><\/p>\n

First articulated in 1975 by British economist Charles Goodhart,<\/a> it holds that that \u201cwhen a measure becomes a target, it ceases to be a good measure.<\/a>\u201d He was writing about monetary policy. But the lesson applies to every case where metrics and the inputs used to support them are separated from important business outcomes.<\/p>\n

And that is exactly the problem with how a recently published MIT study<\/a> chose to frame Gen AI\u2019s ROI.<\/p>\n

Businesses can hit every KPI they know how to measure today and still end up with nothing the market will want tomorrow. That was the Soviet Nail Factory\u2019s fate.<\/p>\n

Only the modern business penalty isn\u2019t Siberia. It\u2019s irrelevance.<\/p>\n

Gen AI In Real Life<\/strong><\/p>\n

Recently, MIT\u2019s much-publicized study<\/a>, The Gen AI Divide, reported that 95% of Gen AI projects fail, and that enterprises see no ROI, as in zero.<\/p>\n

That\u2019s because the payback, according to the report, should be measured in \u201cmillions of direct dollar reductions in external spend,\u201d and in a relatively short space of time. That expectation is the modern equivalent of Gosplan demanding nails by weight. It separates what\u2019s being measured from the outcome.<\/p>\n

That makes the MIT report narrative one that suggests Gen AI is floundering. The evidence over the course of the last 18 months says otherwise.<\/p>\n

Each month since March of 2024, PYMNTS Intelligence has fielded a monthly study to enterprise CFOs, CPOs and COOs to benchmark the Gen AI sentiment, use cases and ROI impacts from the technology over time. We have more than 1,000 unique observations and 100 thousand data points from companies with $1 billion or more in annual revenue. I am not familiar with any other study that systematically measures the impact of Gen AI at the enterprise level over time.<\/p>\n

Over those 18 months, we find that enterprise CFOs, CPOs and COOs view Gen AI as an integral part of their \u201cdigital operating systems.\u201d Over that time, we have observed a steady shift in the use of Gen AI from simplifying routine tasks to becoming an embedded part of strategic functions within their firms<\/a>.<\/p>\n

To take a few examples. CFOs are using Gen AI to model complex financial scenarios, analyze working capital positions and detect anomalies in millions of transactions. Chief Product Officers are using it to automate RFPs, negotiate contracts with better visibility into supplier performance and model risk. Chief Operating Officers are embedding it into logistics, quality assurance and workforce planning. And getting feedback about it all in real time.<\/p>\n

Nearly all (96%) executives at the enterprise level in our studies report favorable positive results, up significantly from even this time a year ago. That\u2019s even though the technology is still in its very earliest innings of potential. And by everyone\u2019s admission still has a long way to go.<\/p>\n

Around the Enterprise Gen AI World in 18 Months<\/strong><\/p>\n

A year and a half ago, we saw enterprise execs using Gen AI with lower-value activities, producing better emails, crisper report summaries and more accurate meeting notes without the hassle of taking them. It was the most logical, low-risk entry point to get familiar with the technology.<\/p>\n

In a relatively short space of time, today we find a 3x increase in enterprise executives saying Gen AI is highly effective for product development and improvement and a 40% increase in the share who find it highly effective for improving workflow management and internal processes. We also see a 28% increase in its effectiveness to manage and monitor cybersecurity activities.<\/p>\n

\"\"<\/p>\n

We also find executives at some of the largest companies in the U.S. reporting a strong, or what I might even call compelling, positive impact of Gen AI across the enterprise, with 90% of enterprise chiefs citing a positive impact on their customer experience<\/a>, and more than three quarters citing a positive impact on their competitive position.<\/p>\n

\"\"<\/p>\n

When it comes to measuring the impact in the more classic dollars-and-cents terms, as I mentioned, 96% report a slight-to-strong ROI on their investments so far, with 28% reporting a very strong business case to support their Gen AI deployments. In just the last six months, the share who will increase their investments in the technology has more than tripled.<\/p>\n

\"\"<\/p>\n

We also find that most enterprise executives have realistic expectations for when they expect positive payback from their investments<\/a>. Not the millions and short-term payback chosen as success metrics by the MIT report. But something that is more aligned with the impact of Gen AI on the business, in ways that are harder to measure precisely right now.<\/p>\n

More than eight in 10 of the more than 1000 enterprise execs we studied believe it could take between three and ten years, depending upon the use case and regulatory and compliance requirements, to generate a meaningful payback.<\/p>\n

These enterprise executives also understand that big-\u201cT\u201d transformation doesn\u2019t usually happen on a predictable timetable, nor with the expectation of an immediate or direct payback \u201cin the millions.\u201d<\/p>\n

It didn\u2019t happen with the commercial internet in the 1990s. It didn\u2019t happen with eCommerce in the 2000s. And it didn\u2019t happen with cloud technology in the 2010s.<\/p>\n

It\u2019s unrealistic to think that will happen with Gen AI today.<\/p>\n

But that hasn\u2019t stopped executives who see its enormous potential from getting their hands dirty and spending money on the tech. Rather than being stressed about the direct financial returns they may gain today, they\u2019re more focused on what they might lose if they sit on the sidelines.<\/p>\n

How Not to Fall into the Gen AI ROI Trap<\/strong><\/p>\n

Gen AI\u2019s value is often the clearest when you look beyond the spreadsheet to outcomes that are sometimes too early to fully measure, but whose potential to reshape how entire industries operate is undeniably powerful.<\/p>\n

Take Unlearn<\/a>, founded by two physicists who saw a path to using Gen AI to reinvent clinical trials. For nearly a decade, they\u2019ve used it to build digital twins that replace traditional placebo groups. Their first focus was on Alzheimer\u2019s and ALS, diseases for which patients want the real deal, not sugar pills. With AI-generated placebos, more participants get the actual drug, trials finish faster and drug development costs drop by billions. The result is measured in life-saving therapies that reach patients years earlier and the scientific evidence<\/a> and Big Pharma backing to support their claims.<\/p>\n

Unlearn\u2019s founders say that it\u2019s the most significant change to trial protocols in 75 years. Yet under MIT\u2019s framework, none of this would count toward ROI because the payoff is still a work in process, even as the impact today is extraordinary. Few would regard this as a \u00a0Gen AI failure.<\/p>\n

Or consider Gen AI transcription in doctors\u2019 offices.<\/p>\n

Gen AI listens so physicians don\u2019t have to type. Doctors can focus on their patients instead of their keyboards. Better conversations happen and subtler patient signals are picked up. Doctors reclaim time and reduce burnout while patients feel heard. The ROI isn\u2019t only in hours saved or more patients that can be seen in a day. It\u2019s in healthier outcomes and better patient\/doctor relationships<\/a>. It\u2019s pretty hard to fit that into a direct-dollar-reduction-in-external-savings template. Yet anyone who\u2019s ever spent time talking to the back of their doctor\u2019s head during an examination clearly understands the value.<\/p>\n

In the business setting, we see improvements in how Gen AI is strengthening cash forecasting across the important middle-market business segment,<\/a> defined as those with between $50 million and $1 billion in annual sales.<\/p>\n

PYMNTS Intelligence\u2019s 2025\/2026 Working Capital Index report, which will be published at the end of September and is based on a collaboration with Visa Commercial Solutions, finds that cash positions are more visible, forecasts are more accurate and working capital use cases are more strategic with the help of Gen AI. And that\u2019s across the entirety of the nearly 1500 CFOs and Treasurers studied. For companies that live or die by how they manage liquidity, Gen AI is becoming the rising tide lifting all boats.<\/p>\n

Then there is the next level of AI. Agents built to simplify complex transactions<\/a>. Today, business often requires navigating endless workflows, systems and approvals. Agents collapse that into a conversation. Time saved. Friction reduced. Outcomes improved. By MIT\u2019s definition, those gains wouldn\u2019t count as ROI because they don\u2019t immediately show up in million dollar savings on a balance sheet. But that\u2019s the misstep. ROI here is speed, clarity and satisfaction. The advantages that business leaders understand create momentum and market share long before they hit the bottom line.<\/a><\/p>\n

These aren\u2019t isolated anecdotes.<\/p>\n

Every major enterprise earnings call in Q2 included specifics about Gen AI, from sharper forecasting to more efficient procurement to productivity gains across operations to better risk management. Companies are using it to do more with fewer people, calling out headcount shifts and efficiency gains as tangible proof points.<\/a> They are investing in Gen AI not as an experiment, but as the foundation for applying the most transformative technology in modern business history to improve how they do business.<\/p>\n

Measuring the Return on Learning<\/strong><\/p>\n

The MIT study suggests that employees don\u2019t understand Gen AI and that this contributes to the lack of a stronger ROI. But no one can master a transformative technology without using it, and a lot. The only way to understand Gen AI is to experiment, test and learn.<\/a><\/p>\n

The irony is that the more transformative the technology, the harder it is to measure ROI in the moment. The internet didn\u2019t prove its worth in quarterly sales. Cloud computing didn\u2019t justify its price tag in its first quarter. The return came as companies built the infrastructure, skills and confidence to use them at scale.<\/p>\n

Gen AI is no different. An important component of ROI is what companies learn by using it.<\/a> Each experiment shortens the distance between analysis and action. Each iteration makes teams more fluent in how to work differently using it.<\/p>\n

That\u2019s why traditional ROI models collapse when applied to Gen AI right now. They assume stable inputs, when the technology is redefining what can be measured and how. They ignore the externalities from learning, including from mistakes.<\/p>\n

Companies reporting the most success don\u2019t see learning as they go as a waste of time or a drawback. They count it as an important part of the transformation process.<\/p>\n

Managing What You Measure<\/strong><\/p>\n

Waiting for Gen AI to show a neat ROI before investing<\/a> is like waiting for the internet to prove it could drive retail sales before building an eCommerce strategy. In the mid-1990s, online sales barely registered compared to brick-and-mortar stores. By traditional ROI math, eCommerce looked like a waste of time and money, and for years. But what mattered wasn\u2019t the sales it generated in the short term. It was how it would rewrite the rules of retail.<\/p>\n

Those who waited for online sales to be \u201cbig enough\u201d to justify the investment found themselves chasing Amazon. Some still do. Many aren\u2019t around anymore as foot traffic plummeted.<\/p>\n

Cloud computing followed the same arc. In its early years, companies measured ROI by the cost savings of getting rid of servers. By that metric, the payoff looked modest. But the real return wasn\u2019t a cheaper IT department. It was agility. With cloud, a startup could scale globally without buying hardware. Enterprises could launch new products without waiting months for infrastructure. Cloud made experimentation cheap, failure less costly and global scale possible in weeks, not years. Those who measured only cost savings missed the revolution happening under their noses.<\/p>\n

Gen AI follows the same trajectory, although arguably much faster<\/a>. Like eCommerce and cloud, it redefines the inputs that matter. The laggards will keep looking for a percentage savings in costs or headcount. The leaders will measure how fast their organizations can learn, adapt and build fluency. They see the return not in the pilot itself, but in the organizational muscle that comes from learning how to think and act differently.<\/p>\n

When Everything Is a Nail <\/strong><\/p>\n

Managers like to repeat Peter Drucker\u2019s line that what gets measured gets managed. But that only works if business leaders are measuring the right thing. The Soviet Nail Factory proved the danger of getting it wrong. Quotas by volume produced flimsy nails, quotas by weight produced nails no one could use. Metrics were met, but value was destroyed.<\/p>\n

That\u2019s why the leaders funding Gen AI aren\u2019t hung up on short-term ROI. They see its edge strategically, not just tactically. They understand that the value right now isn\u2019t visible on the spreadsheets that show a headcount reduction or a faster cycle time. It\u2019s in building the capacity to think and act differently, something that\u2019s much harder to measure quarter to quarter.<\/p>\n

It\u2019s the risk with dismissing Gen AI as hype. Measuring output is easy. The harder, more important task is knowing which inputs matter at the same time the rules of the game are being rewritten and the technology is evolving in real time. MIT\u2019s insistence on millions in direct dollar savings is the modern equivalent of Gosplan\u2019s quotas: an artificial target built on inputs that are still being defined.<\/p>\n

But the real danger with Gen AI isn\u2019t hype. It\u2019s business executives misreading the inputs. It\u2019s tempting to be skeptical, especially when Daron Acemoglu, a 2024 Nobel laureate in economics<\/a> says that Gen AI, at best, will automate profitably only 5% of all tasks and have a relatively modest impact on GDP and productivity.\u00a0 It\u2019s easy to shrug off Gen AI as \u201cLinkedIn hype,\u201d just like one of MIT\u2019s survey respondents was quoted as saying.<\/p>\n

Fortunately, most executives have recognized the importance of investing and learning about Gen AI because they have seen enough evidence of its power to make them confident about its long-term importance for their businesses.<\/p>\n

Those who don\u2019t risk living in a future already designed by those who didn\u2019t wait, using it to change how business gets done.<\/p>\n

What\u2019s your take?<\/p>\n

\"\"\/<\/a><\/p>\n

Until NEXT time.<\/p>\n

Join the more than 15,000 subscribers who\u2019ve already said yes to what\u2019s NEXT.<\/strong><\/a><\/p>\n

Image credit: Cartoon appeared in the Soviet magazine Krokodil in 1957.<\/p>\n","protected":false},"excerpt":{"rendered":"Everything you always wanted to know about how to measure the return on investment of generative AI (and…\n","protected":false},"author":2,"featured_media":41047,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[261],"tags":[291,289,290,18,30542,19,17,31175,31176,13060,5,1350,82],"class_list":{"0":"post-41046","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-artificialintelligence","11":"tag-eire","12":"tag-gen-ai","13":"tag-ie","14":"tag-ireland","15":"tag-karen-webster","16":"tag-klw-commentary","17":"tag-main-feature","18":"tag-news","19":"tag-pymnts-intelligence","20":"tag-technology"},"share_on_mastodon":{"url":"","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/posts\/41046","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/comments?post=41046"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/posts\/41046\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/media\/41047"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/media?parent=41046"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/categories?post=41046"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/tags?post=41046"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}