demonstrated, as a proof-of-concept, how an artificial intelligence (AI) assistant in the form of Microsoft Copilot could be used to compromise an email system<\/a> in a way that not only provides access to sensitive information but also creates messages that could be sent to unsuspecting additional targets.<\/p>\nMerium Khalid, director of AI and Automation for the Office of the CTO at Barracuda Networks, said a cyberattack against an email system that once required a fair amount of expertise is now relatively trivial to execute. For example, a prompt could be used by a malicious actor to gain a sense of the organizational structure, especially any ongoing conversations that might expose worthwhile pivots. The malicious actor can also create an inbox rule that forwards any sign-in notifications into the \u201cDeleted Items\u201d folder to ensure an end user does not receive any notifications for unusual sign-ins.<\/p>\n
From there, it becomes possible to send an email from the employee to a CEO that includes a link to an invoice for them to approve. Once approved, the AI agent mimics the way the CEO communicates to then forward an email to finance. Because the message came from the CEO\u2019s real mailbox, passed every authentication check, referenced a real in-flight transaction, and matched the CEO\u2019s usual tone, there is nothing for traditional email security platforms to flag. The PoC added a forwarding rule in the CEO\u2019s mailbox to ensure the finance team\u2019s reply to confirm the bank change was never seen.<\/p>\n
Finally, attackers used Copilot to rapidly locate messages associated with the fraud and remove them before any manual review could be completed.<\/p>\n
While the email attack created by Barracuda Networks is a PoC, there are examples of similar attacks being made using AI tools embedded within email systems, said Khalid. The challenge, of course, is these types of attacks can now be made in an instant, so it\u2019s now more critical than ever for organizations to be able to invoke some type of kill switch within an email workflow, she added.<\/p>\n
At the same time, organizations also need to both train end users to better recognize these types of attacks and continuously monitor messages for unusual activity of any kind, noted Khalid.<\/p>\n
It\u2019s not clear how pervasive AI tools and agents might already have been hijacked, but the probability that similar business email compromises will occur in the age of AI is high. An AI assistant effectively acts as a knowledgeable insider, helping attackers identify sensitive information, understand organizational relationships, target privileged users and execute fraudulent transactions more efficiently, noted Khalid.<\/p>\n
Hopefully, AI agents will not force organizations to find alternative means for managing workflows that may not be so easily compromised. The challenge and the opportunity now is to find a way to securely and safely add AI agents to existing workflows that would otherwise be difficult to replace. Unfortunately, however, it\u2019s likely there will be many hard lessons of what not to do before organizations revisit their existing approaches to email security.<\/p>\n
Frequently Asked QuestionsHow could attackers use Microsoft Copilot to compromise email?<\/p>\n
A malicious actor could use the assistant to search messages, understand organizational relationships, identify sensitive conversations and create mailbox rules that conceal suspicious activity.<\/p>\n
How can organizations reduce the risk of AI-powered email fraud?<\/p>\n
They should monitor unusual mailbox activity, restrict AI agent permissions, require independent verification for payment changes, train employees and maintain a way to immediately disable compromised AI workflows.<\/p>\n
Was this a real attack or a demonstration?<\/p>\n
Barracuda\u2019s scenario was a proof of concept, although the company said similar attacks involving AI tools embedded in email systems have already been observed.<\/p>\n","protected":false},"excerpt":{"rendered":"TL;DR \u2014 Key Takeaways Barracuda demonstrated how a compromised Microsoft Copilot assistant could analyze emails, identify valuable targets…\n","protected":false},"author":2,"featured_media":129162,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[15193,37131,64569,64570,420,7853,42474,64571,42475,416,64572,28580,64573,64574,10659,64575,25631,17775,21701,64576,320,7852,64577],"class_list":["post-129161","post","type-post","status-publish","format-standard","has-post-thumbnail","category-microsoft","tag-ai-agent-governance","tag-ai-assistant-security","tag-ai-email-attacks","tag-ai-powered-phishing","tag-azure","tag-azure-copilot","tag-barracuda-networks","tag-bec-attacks","tag-business-email-compromise","tag-copilot","tag-copilot-vulnerabilities","tag-cybersecurity-awareness","tag-email-account-takeover","tag-email-forwarding-rules","tag-email-security","tag-executive-impersonation","tag-financial-fraud","tag-generative-ai-security","tag-insider-threats","tag-mailbox-compromise","tag-microsoft","tag-microsoft-copilot","tag-microsoft-copilot-security"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/129161","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=129161"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/129161\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/129162"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=129161"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=129161"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=129161"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}