TL;DR — Key Takeaways
Barracuda demonstrated how a compromised Microsoft Copilot assistant could analyze emails, identify valuable targets and manipulate mailbox settings.
Attackers could impersonate executives using real accounts, authentic conversations and AI-generated messages that match their normal communication style.
Fraudulent emails sent from genuine mailboxes may bypass traditional authentication and email security controls.
Organizations need rapid kill switches, continuous mailbox monitoring, stronger AI governance and independent verification for financial requests.
Barracuda Networks today demonstrated, as a proof-of-concept, how an artificial intelligence (AI) assistant in the form of Microsoft Copilot could be used to compromise an email system in a way that not only provides access to sensitive information but also creates messages that could be sent to unsuspecting additional targets.
Merium Khalid, director of AI and Automation for the Office of the CTO at Barracuda Networks, said a cyberattack against an email system that once required a fair amount of expertise is now relatively trivial to execute. For example, a prompt could be used by a malicious actor to gain a sense of the organizational structure, especially any ongoing conversations that might expose worthwhile pivots. The malicious actor can also create an inbox rule that forwards any sign-in notifications into the “Deleted Items” folder to ensure an end user does not receive any notifications for unusual sign-ins.
From there, it becomes possible to send an email from the employee to a CEO that includes a link to an invoice for them to approve. Once approved, the AI agent mimics the way the CEO communicates to then forward an email to finance. Because the message came from the CEO’s real mailbox, passed every authentication check, referenced a real in-flight transaction, and matched the CEO’s usual tone, there is nothing for traditional email security platforms to flag. The PoC added a forwarding rule in the CEO’s mailbox to ensure the finance team’s reply to confirm the bank change was never seen.
Finally, attackers used Copilot to rapidly locate messages associated with the fraud and remove them before any manual review could be completed.
While the email attack created by Barracuda Networks is a PoC, there are examples of similar attacks being made using AI tools embedded within email systems, said Khalid. The challenge, of course, is these types of attacks can now be made in an instant, so it’s now more critical than ever for organizations to be able to invoke some type of kill switch within an email workflow, she added.
At the same time, organizations also need to both train end users to better recognize these types of attacks and continuously monitor messages for unusual activity of any kind, noted Khalid.
It’s not clear how pervasive AI tools and agents might already have been hijacked, but the probability that similar business email compromises will occur in the age of AI is high. An AI assistant effectively acts as a knowledgeable insider, helping attackers identify sensitive information, understand organizational relationships, target privileged users and execute fraudulent transactions more efficiently, noted Khalid.
Hopefully, AI agents will not force organizations to find alternative means for managing workflows that may not be so easily compromised. The challenge and the opportunity now is to find a way to securely and safely add AI agents to existing workflows that would otherwise be difficult to replace. Unfortunately, however, it’s likely there will be many hard lessons of what not to do before organizations revisit their existing approaches to email security.
Frequently Asked QuestionsHow could attackers use Microsoft Copilot to compromise email?
A malicious actor could use the assistant to search messages, understand organizational relationships, identify sensitive conversations and create mailbox rules that conceal suspicious activity.
How can organizations reduce the risk of AI-powered email fraud?
They should monitor unusual mailbox activity, restrict AI agent permissions, require independent verification for payment changes, train employees and maintain a way to immediately disable compromised AI workflows.
Was this a real attack or a demonstration?
Barracuda’s scenario was a proof of concept, although the company said similar attacks involving AI tools embedded in email systems have already been observed.