{"id":60454,"date":"2026-05-25T22:10:28","date_gmt":"2026-05-25T22:10:28","guid":{"rendered":"https:\/\/www.europesays.com\/canada\/60454\/"},"modified":"2026-05-25T22:10:28","modified_gmt":"2026-05-25T22:10:28","slug":"mark-carney-thinks-ai-will-save-money-it-will-also-cost-lives","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/canada\/60454\/","title":{"rendered":"Mark Carney Thinks AI Will Save Money. It Will Also Cost Lives"},"content":{"rendered":"

The Mark Carney government has made \u201cdeploying AI at scale<\/a>\u201d a cornerstone of its attempt to make government more productive and slash costs by cutting 28,000 jobs<\/a> by 2029. <\/p>\n

Announcements, Events & more from Tyee and select partners<\/p>\n

The Tyee Is Recruiting Our Next Editor-in-Chief<\/a>
\n\"The<\/a><\/p>\n

Founding editor David Beers is passing the baton to a new leader. Is it you?<\/p>\n

The goal is to achieve savings of $60 billion<\/a> over several years.<\/p>\n

There are many reasons to be skeptical of the government\u2019s AI strategy. Savings projections resulting from digitalization should be taken with a grain of salt. For example, the Phoenix pay system, designed to automate the federal payroll, was supposed to save $70 million per year. Instead, unable to deal with the complexity of paying hundreds of thousands of public servants, it has cost the federal government $4.34 billion and climbing<\/a> to try to fix it.<\/p>\n

However, unrealized savings are the least of the concerns coming from the government\u2019s wholehearted embrace of AI, including algorithmic-based tools. Deploying these technologies as cost-cutting measures will not only result in worse service for Canadians, but also put lives at risk \u2014 as has already happened here and in other countries.<\/p>\n

If the federal government is intent on exploring the use of AI (however it is defined) in government, it should do so not as a cost-cutting measure, but only after careful, case-by-case deliberation that pays close attention to how this (or any) technology interacts with the people using, and affected by, the tech in question.<\/p>\n


\n <\/p>\n

Two forms of AI<\/p>\n

The problems begin with the technologies themselves. Simplifying greatly, focus on two general forms of AI.<\/p>\n

The first, \u201cgenerative AI\u201d such as ChatGPT produces probabilistic output predicting what the next word is likely to be in a sequence based on its training data. It produces patterns that look like human thought, but it\u2019s just repeating patterns in its data. As such, it\u2019s prone to producing \u201challucinations,\u201d which can involve presenting false information as true.<\/p>\n

These are not technically incorrect outputs per se because the program is simply doing what it\u2019s designed to do: provide probabilistically determined strings of words and sentences. The fact that this problem cannot be fixed<\/a> means that its output can never be fully trusted.<\/p>\n

The second form is \u201cdiscriminative AI,\u201d such as decision-making algorithms that produce options and targets by identifying patterns in data. Such programs are only as good as the data provided and their algorithms \u2014 both of which are created by humans and therefore share their biases and fallibilities.<\/p>\n

Two recent Canadian cases reveal that governments\u2019 experimentation with generative AI and algorithmic decision-making in delivering immigration and social-assistance programs has already resulted in serious negative consequences.<\/p>\n

In the first case, Immigration, Refugees and Citizenship Canada acknowledged using generative AI<\/a> to reject an application for permanent residence on the grounds the applicant\u2019s job duties didn\u2019t match her claimed Canadian job experience. However, the AI tool erroneously generated the applicant\u2019s current job duties, which means the algorithm wrongly rejected the application.<\/p>\n

In the second case, Quebec launched an AI-driven overhaul in 2025 of its social-assistance system called Project UNIR, which uses algorithms<\/a> to help determine eligibility for financial assistance. The project eliminated the previous \u201cassigned agents\u201d who worked with clients from the beginning to the end of their files. It now divides tasks in each applicant\u2019s file among officials working in different regions.<\/p>\n

One man killed himself after being given incorrect information by staff saying he was ineligible for assistance \u2014 a mistake exacerbated by the lack of a human agent who knew the context of the man\u2019s file. Other people calling the system have expressed suicidal thoughts because of its administrative delays and document losses.<\/p>\n

What makes these two cases \u2014 as well as the three deaths from listeria<\/a> in 2024 due to the Canadian Food Inspection Agency\u2019s reliance on an algorithm and bad data to determine which manufacturing facilities to investigate \u2014 so frustrating is that they mirror tragic events earlier and elsewhere.<\/p>\n

In 2016, Australia introduced an automated debt-recovery program to identify potential welfare fraud. The program, known as Robodebt and based on an algorithm, was so riddled with errors<\/a> that it wrongly identified 450,000 individuals as being involved in fraud. Robodebt sparked at least three suicides<\/a>, police investigations, a royal commission and an agreement for the Australian government to pay AU$475 million in compensation to victims.<\/p>\n

The Australian government\u2019s ill-fated, costly experiment with this algorithmic decision-making and the Canadian cases hold significant lessons on the consequences of turning to algorithms to operate and manage public services while cutting back on frontline public servants.<\/p>\n


\n <\/p>\n

Such debacles share important similarities, as we explored in our 2023 book<\/a> The New Knowledge: Information, Data and the Remaking of Global Power.<\/p>\n

The first involves the quality of the technology and the consequences of automating public services.<\/p>\n

The reliance on generative AI to create actionable reports is itself a problem because it can be unable to deal with the complexity of real-world cases, while making it difficult to impossible for human caseworkers to intervene to correct problems.<\/p>\n

It\u2019s therefore more difficult for clients to figure out what\u2019s happening and why a decision was made \u2014 a problem that\u2019s exacerbated as anxious clients are unable to reach human agents via jammed phone lines. The Quebec government has spent millions on a private firm to handle the extra phone calls.<\/p>\n

The second regards the role of workers using or affected by these technologies. To mitigate the harms caused by hallucinations and algorithmic decision-making, governments have tended to embrace a \u201chuman in the loop<\/a>\u201d strategy, ensuring people participate in the operation and supervision of algorithm-driven systems.<\/p>\n

However, a human-in-the-loop rule is not sufficient to guard against errors or prevent harms. The very presence of the technology affects how people do their jobs.<\/p>\n

The shift to automated programs often constrains or even prevents frontline staff from using their experience and expertise to make decisions. Scholars refer to this as the rise of \u201c
\nscreen-level bureaucracy<\/a>\u201d because bureaucracy does not disappear but rather changes form and becomes less accountable as algorithmic decisions are typically delivered opaquely via private-sector technology.<\/p>\n

What\u2019s more, people\u2019s well-documented tendency to treat computer outputs as authoritative is supercharged when workers are asked to do more with less, giving them less time to perform due diligence on these algorithmically generated outputs.<\/p>\n

Finally, researchers are increasingly concerned that reliance on AI technologies will lead to deskilling<\/a>, meaning civil servants will lose the ability to develop the skills and expertise needed to catch AI errors. <\/p>\n

This is another reason why the human-in-the-loop strategy is deeply flawed. The more you use these technologies, the worse your overall skills will become.<\/p>\n

Beyond the human-in-the-loop strategy, governments are attempting other mitigating factors.<\/p>\n

For example, Immigration, Refugees and Citizenship Canada says in its artificial intelligence strategy<\/a> that the department uses AI for administrative tasks such as summarizing and producing documents but that those tools do not themselves reject or recommend rejecting applications. IRCC also rates AI-delivered document summary and production as low risk, while it rates AI informing decision-makers as medium risk.<\/p>\n

However, IRCC\u2019s distinction between low and medium risk is not useful if human decisions on files, which presumably will feed into future consequential decisions, are based on erroneous information from AI tools.<\/p>\n

For governments, coming to terms with this reality means recognizing that making such technologies work requires human analysis and review at every step.<\/p>\n

Far from cutting labour costs, only a well-resourced and skilled workforce can adequately manage these technologies. If workers are not given sufficient time and power to review algorithmically generated outputs, then system breakdown and worse service is all too likely.<\/p>\n

The Carney government has placed its faith in AI to deliver low-cost government services. These technologies, used only in specific circumstances, may offer some benefits to Canadians.<\/p>\n

However, not only does starting with your preferred solution foreclose other, potentially better options; neither AI technology as it currently exists nor the many examples of what happens when such technologies are introduced augur anything but a slow-motion disaster for Canadians and the public service. \"<\/p>\n","protected":false},"excerpt":{"rendered":"The Mark Carney government has made \u201cdeploying AI at scale\u201d a cornerstone of its attempt to make government…\n","protected":false},"author":2,"featured_media":60455,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[98],"tags":[26160,26161,26162,7728,26167,26168,3855,4358,111,26163,26164,6678,26165,26166],"class_list":["post-60454","post","type-post","status-publish","format-standard","has-post-thumbnail","category-mark-carney","tag-ai-hallucinations","tag-ai-human-in-the-loop","tag-canada-ai-strategy","tag-canadian-food-inspection-agency","tag-data-and-the-remaking-of-global-power","tag-discriminative-ai","tag-generative-ai","tag-immigration","tag-mark-carney","tag-phoenix-pay-system","tag-project-unir","tag-refugees-and-citizenship-canada","tag-robodebt-australia","tag-the-new-knowledge-information"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/canada\/wp-json\/wp\/v2\/posts\/60454","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/canada\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/canada\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/canada\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/canada\/wp-json\/wp\/v2\/comments?post=60454"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/canada\/wp-json\/wp\/v2\/posts\/60454\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/canada\/wp-json\/wp\/v2\/media\/60455"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/canada\/wp-json\/wp\/v2\/media?parent=60454"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/canada\/wp-json\/wp\/v2\/categories?post=60454"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/canada\/wp-json\/wp\/v2\/tags?post=60454"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}