Gray Swan cofounders Matt Fredrikson and Zico Kolter started the company in 2023 to help AI juggernauts like OpenAI and Anthropic pressure test their models for safety vulnerabilities.<\/p>\n
Gray Swan <\/p>\n
Last spring, Kameron Bettridge participated in a security challenge hosted by AI startup Gray Swan. The objective: convince AI models from companies like OpenAI and Anthropic to behave in nefarious ways before they\u2019re released to the world. That included persuading the models to leak sensitive data like medical records and spit out copyrighted information like the full lyrics of Hotel California. <\/p>\n
At first Bettridge, a 23-year-old security engineer at gaming company Blizzard Entertainment, was jailbreaking models for fun. \u201cI’ve never been a true supporter of AI fully,\u201d he says. \u201cSo just seeing the model fail was a funny thing to me sometimes.\u201d<\/p>\n
In almost a year, Bettridge has competed in more than 1,000 challenges via Arena\u2014 a hub run by startup Gray Swan that some 15,000 security professionals from all across the world use to \u201cred team<\/a>\u201d AI systems like Anthropic\u2019s Claude Mythos and OpenAI\u2019s GPT-5, finding and fixing vulnerabilities before they can be exploited. And he\u2019s made $10,000 doing it. <\/p>\n It\u2019s not a lot for a highly paid software engineer. But as AI became ubiquitous, Bettridge realized just how important it is to test the limits of these AI models. The technology has been used to plan mass shootings, steal money and create illegal child sexual abuse material. \u201cNow we have very strong models that anyone can access from anywhere in the world, which is a scary thought,\u201d Bettridge says. \u201cPeople are genuinely trying to use this for harmful things.\u201d <\/p>\n Founded in 2023 by Carnegie Mellon University professors Matt Fredrikson and Zico Kolter, Gray Swan has become the go-to security provider for a who\u2019s who of frontier labs: OpenAI, Anthropic, Google Deepmind, Meta, xAI and ByteDance. The startup has been cited in 11 frontier model system cards including GPT-5 and Mythos \u2014 documents that list the risks an AI model poses and safety measures taken to prevent them. <\/p>\n Now, it\u2019s raised $40 million in Series A funding co-led by Wing VC and Madrona with participation from Snowflake Ventures, Hudson River Trading, and Samsung Next, bringing its valuation to $200 million. It already has 20 enterprise customers, but the funds will help it sell to more businesses that need to secure their own AI products. <\/p>\n While Gray Swan runs Arena, (not to be confused with LMArena that benchmarks models based on performance), that isn\u2019t its primary product. But it uses the data from Arena\u2019s human red-teamers to train its AI agent called Shade that actively looks for vulnerabilities by continuously attacking a system in different ways, and Cygnal, software that monitors an AI model\u2019s prompts and outputs to block it from generating harmful responses and accessing tools it shouldn\u2019t. That human data is its edge, allowing Gray Swan to throw hackers\u2019 most sophisticated attacks against increasingly capable AI models. <\/p>\n \u201cAgents are now much smarter,\u201d says chief scientist and cofounder Kolter, who also sits on the board of OpenAI Foundation. \u201cThey are looking for prompt injections. They’re trying to defeat these things. They’re not trying to stumble upon these things.\u201d <\/p>\n The Pittsburgh-based startup gained an early foothold among the biggest AI labs thanks to its founder\u2019s hacker pedigree. The duo began researching the safety risks posed by AI systems years before the generative AI wave. In 2023, they discovered what was dubbed \u201cthe mother of all jailbreaks<\/a>\u201d \u2014 that attaching a string of random characters to a prompt could bypass safety filters on models built by OpenAI, Anthropic, Meta and Google (it\u2019s since been fixed). That sparked the idea to start Gray Swan. <\/p>\n Less than a month after the company launched, OpenAI became its first customer, using its technology to jailbreak its family of o1 models, testing whether they generate violent content and malicious code. In 2024, Kolter was appointed to the OpenAI Foundation\u2019s board, where he oversees major model releases as chair of the safety and security committee. <\/p>\n \u201cThey were thinking about model security when it just didn’t matter,\u201d says Wing VC partner Jake Flomenberg. \u201cThey had literally been spending their entire professional life working on this very problem from an academic setting. And so they were both sort of at the right place with their thinking and research for this big sea change.\u201d <\/p>\n While frontier labs make up a majority of its revenue, Gray Swan is increasingly appealing to large enterprises. Snowflake uses Gray Swan\u2019s software to pressure test its coding agent, Cortex Code and its general purpose agent, Snowflake Intelligence, which it sells to customers, says Anupam Datta, a principal research scientist at Snowflake. In one scenario, Gray Swan\u2019s software looks for malicious prompts hidden within external websites or tools Snowflake\u2019s agents might access to complete a task. These prompts could instruct the agent to send internal proprietary data, such as information about the company\u2019s earnings, to an email address managed by an adversary. \u201cGray Swan can guard against very subtle kinds of attacks,\u201d Datta says. <\/p>\n As AI systems become more intelligent, jailbreaking them will require more complexity and nuance, CEO Fredrikson says. Agents find new loopholes to exploit. Because these systems interact with a web of tools, the \u201csurface area\u201d of attacks has become bigger. <\/p>\n \u201cThe one thing you can rely on is that there are going to be surprises,\u201d Fredrikson says. \u201cThese systems can create new attack surfaces that we’re not even thinking about today that aren’t obvious.\u201d <\/p>\n MORE FROM FORBES<\/p>\n ForbesSarah Guo Bet Everything On AI Pre-ChatGPT. Now She\u2019s One Of The World\u2019s Top InvestorsBy Rashi Shrivastava<\/a>ForbesThrive And Sequoia Back Pace With $46 Million To Automate Insurance\u2019s Back OfficeBy Anna Tong<\/a>ForbesElon Musk Interview: OpenAI Verdict Sets A \u2018Dangerous Precedent\u2019By Richard Nieva<\/a>ForbesVCs’ Payday From SpaceX, OpenAI And Anthropic Could Top All Profits Of The Last DecadeBy Iain Martin<\/a>ForbesWe Have No Clue What Goes On Inside AI’s Brain. This $1.25 Billion Startup Is Trying To Find OutBy Rashi Shrivastava<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"Gray Swan cofounders Matt Fredrikson and Zico Kolter started the company in 2023 to help AI juggernauts like…\n","protected":false},"author":2,"featured_media":54088,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[24,53,3154,182,31751,31750,31752,157,8922,225,314,31749],"class_list":["post-54087","post","type-post","status-publish","format-standard","has-post-thumbnail","category-anthropic","tag-ai","tag-anthropic","tag-anthropic-claude","tag-claude","tag-gray-swan-arena","tag-jailbreaking","tag-op","tag-openai","tag-red-teaming","tag-safety","tag-security","tag-zico-kolter"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/54087","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=54087"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/54087\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/54088"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=54087"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=54087"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=54087"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}