It\u2019s like a new telling of the \u201cTortoise and the Hare\u201d: A group of experienced software engineers entered into an experiment where they were tasked with completing some of their work with the help of AI tools. Thinking like the speedy hare, the developers expected AI to expedite their work and increase productivity. Instead, the technology slowed them down more. The AI-free tortoise approach, in the context of the experiment, would have been faster.\u00a0<\/p>\n
The results of this experiment, published in a study<\/a> this month, came as a surprise to the software developers tasked with using AI\u2014and to the study\u2019s authors, Joel Becker and Nate Rush, technical staff members of nonprofit technology research organization Model Evaluation and Threat Research (METR).<\/p>\n The researchers enlisted 16 software developers, who had an average of five years of experience, to conduct 246 tasks, each one a part of projects on which they were already working. For half the tasks, the developers were allowed to use AI tools\u2014most of them selected code editor Cursor Pro or Claude 3.5\/3.7 Sonnet\u2014and for the other half, the developers conducted the tasks on their own.<\/p>\n Believing the AI tools would make them more productive, the software developers predicted the technology would reduce their task completion time by an average of 24%. Instead, AI resulted in their task time ballooning to 19% greater than when they weren\u2019t using the technology.<\/p>\n \u201cWhile I like to believe that my productivity didn\u2019t suffer while using AI for my tasks, it\u2019s not unlikely that it might not have helped me as much as I anticipated or maybe even hampered my efforts,\u201d Philipp Burckhardt, a participant in the study, wrote in a blog post<\/a> about his experience.<\/p>\n Why AI is slowing some workers down<\/strong><\/p>\n So where did the hares veer off the path? The experienced developers, in the midst of their own projects, likely approached their work with plenty of additional context their AI assistants did not have, meaning they had to retrofit their own agenda and problem-solving strategies into the AI\u2019s outputs, which they also spent ample time debugging, according to the study.\u00a0<\/p>\n \u201cThe majority of developers who participated in the study noted that even when they get AI outputs that are generally useful to them\u2014and speak to the fact that AI generally can often do bits of very impressive work, or sort of very impressive work\u2014these developers have to spend a lot of time cleaning up the resulting code to make it actually fit for the project,\u201d study author Rush told Fortune.<\/p>\n Other developers lost time writing prompts for the chatbots or waiting around for the AI to generate results.<\/p>\n The results of the study contradict lofty promises about AI\u2019s ability to transform the economy and workforce, including a 15% boost to U.S. GDP<\/a> by 2035 and eventually a 25% increase in\u00a0productivity<\/a>.\u00a0<\/p>\n But Rush and Becker have shied away from making sweeping claims about what the results of the study mean for the future of AI.<\/p>\n For one, the study\u2019s sample was small and non-generalizable, including only a specialized group of people to whom these AI tools were brand new. The study also measures technology at a specific moment in time, the authors said, not ruling out the possibility that AI tools could be developed in the future that would indeed help developers enhance their workflow.<\/p>\n The purpose of the study was, broadly speaking, to pump the brakes on the torrid implementation of AI in the workplace and elsewhere, acknowledging more data about AI\u2019s actual effects need to be made known and accessible before more decisions are made about its applications.<\/p>\n \u201cSome of the decisions we\u2019re making right now around development and deployment of these systems are potentially very high consequence,\u201d Rush said. \u201cIf we\u2019re going to do that, let\u2019s not just take the obvious answer. Let\u2019s make high-quality measurements.\u201d<\/p>\n AI\u2019s broader impact on productivity<\/strong><\/p>\n Economists have already asserted that METR\u2019s research aligns with broader narratives on AI and productivity. While AI is beginning to chip away at entry-level positions<\/a>, according to LinkedIn<\/a> chief economic opportunity officer Aneesh Raman, it may offer diminishing returns for skilled workers such as experienced software developers.<\/p>\n \u201cFor those people who have already had 20 years, or in this specific example, five years of experience, maybe it\u2019s not their main task that we should look for and force them to start using these tools if they\u2019re already well functioning in the job with their existing work methods,\u201d Anders Humlum, an assistant professor of economics at the University of Chicago\u2019s Booth School of Business, told Fortune.<\/p>\n Humlum has similarly conducted research on AI\u2019s impact on productivity. He found in a working study<\/a> from May that among 25,000 workers in 7,000 workplaces in Denmark\u2014a country with similar AI uptake as the U.S.\u2014productivity improved a modest 3% among employees using the tools.\u00a0<\/p>\n Humlum\u2019s research supports MIT economist and Nobel laureate Daron Acemoglu\u2019s assertion that markets have overestimated productivity gains from AI. Acemoglu argues only 4.6% of tasks within the U.S. economy will be made more efficient with AI.<\/p>\n \u201cIn a rush to automate everything, even the processes that shouldn\u2019t be automated, businesses will waste time and energy and will not get any of the productivity benefits that are promised,\u201d Acemoglu previously wrote for Fortune<\/a>. \u201cThe hard truth is that getting productivity gains from any technology requires organizational adjustment, a range of complementary investments, and improvements in worker skills, via training and on-the-job learning.\u201d<\/p>\n The case of the software developers\u2019 hampered productivity points to this need for critical thought on when AI tools are implemented, Humlum said. While previous research on AI productivity has looked at self-reported data<\/a> or specific and contained tasks<\/a>, data on challenges from skilled workers using the technology complicate the picture.<\/p>\n \u201cIn the real world, many tasks are not as easy as just typing into ChatGPT,\u201d Humlum said. \u201cMany experts have a lot of experience [they\u2019ve] accumulated that is highly beneficial, and we should not just ignore that and give up on that valuable expertise that has been accumulated.\u201d<\/p>\n \u201cI would just take this as a good reminder to be very cautious about when to use these tools,\u201d he added.<\/p>\n","protected":false},"excerpt":{"rendered":"It\u2019s like a new telling of the \u201cTortoise and the Hare\u201d: A group of experienced software engineers entered…\n","protected":false},"author":3,"featured_media":77903,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21],"tags":[691,738,10841,1183,53828,158,67,132,68],"class_list":{"0":"post-77902","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-productivity","11":"tag-research","12":"tag-software-developers","13":"tag-technology","14":"tag-united-states","15":"tag-unitedstates","16":"tag-us"},"share_on_mastodon":{"url":"https:\/\/pubeurope.com\/@us\/114885411619071685","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts\/77902","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/comments?post=77902"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts\/77902\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/media\/77903"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/media?parent=77902"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/categories?post=77902"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/tags?post=77902"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}