\u201cAI is expensive. Let’s be honest about that,\u201d Anand says.<\/p>\n
Growth vs. Safety<\/p>\n
In October 2024, the mother of a teen who died by suicide filed a wrongful death suit<\/a> against Character Technologies, its founders, Google, and Alphabet, alleging the company targeted her son with \u201canthropomorphic, hypersexualized, and frighteningly realistic experiences, while programming [the chatbot] to misrepresent itself as a real person, a licensed psychotherapist, and an adult lover.\u201d At the time, a Character.AI spokesperson told CNBC<\/a> that the company was \u201cheartbroken by the tragic loss\u201d and took \u201cthe safety of our users very seriously.\u201d<\/p>\n The tragic incident put Character.AI under intense scrutiny. Earlier this year, US senators Alex Padilla and Peter Welch wrote<\/a> a letter to several AI companionship platforms, including Character.AI, highlighting concerns about \u201cthe mental health and safety risks posed to young users\u201d of the platforms.<\/p>\n \u201cThe team has been taking this very responsibly for almost a year now,\u201d Anand tells me. \u201cAI is stochastic, it’s kind of hard to always understand what’s coming. So it’s not a one time investment.\u201d<\/p>\n That\u2019s critically important because Character.AI is growing. The startup has 20 million monthly active users who spend, on average, 75 minutes a day chatting with a bot (a \u201ccharacter\u201d in Character.AI parlance). The company\u2019s user base is 55 percent female. More than 50 percent of its users are Gen Z or Gen Alpha. With that growth comes real risk\u2014what is Anand doing to keep his users safe?<\/p>\n \u201c[In] the last six months, we’ve invested a disproportionate amount of resources in being able to serve under 18 differently than over 18, which was not the case last year,\u201d Anand says. \u201cI can’t say, \u2018Oh, I can slap an 18+ label on my app and say use it for NSFW.\u2019 You end up creating a very different app and a different small-scale platform.\u201d<\/p>\n More than 10 of the company\u2019s 70 employees work full-time on trust and safety, Anand tells me. They\u2019re responsible for building safeguards like age verification, separate models for users under 18, and new features such as parental insights<\/a>, which allow parents to see how their teens are using the app.<\/p>\n The under-18 model launched last December. It includes \u201ca narrower set of searchable Characters on the platform,\u201d according to company spokesperson Kathryn Kelly. \u201cFilters have been applied to this set to remove Characters related to sensitive or mature topics.\u201d<\/p>\n But Anand says AI safety will take more than just technical tweaks. \u201cMaking this platform safe is a partnership between regulators, us, and parents,\u201d Anand says. That\u2019s what makes watching his daughter chat with a Character so important. \u201cThis has to stay safe for her.\u201d<\/p>\n Beyond Companionship<\/p>\n The AI companionship market is booming. Consumers worldwide spent $68 million on AI companionship in the first half of this year, a 200 percent increase from last year, according to an estimate cited by CNBC<\/a>. AI startups are gunning for a slice of the market: xAI released<\/a> a creepy, pornified companion in July, and even Microsoft bills<\/a> its Copilot chatbot as an AI companion.<\/p>\n So how does Character.AI stand out in a crowded market? It takes itself out of it entirely.<\/p>\n","protected":false},"excerpt":{"rendered":"\u201cAI is expensive. Let’s be honest about that,\u201d Anand says. Growth vs. Safety In October 2024, the mother…\n","protected":false},"author":3,"featured_media":140062,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21],"tags":[691,738,16028,251,66585,8152,158,67,132,68],"class_list":{"0":"post-140061","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-chatbots","11":"tag-kids","12":"tag-model-behavior","13":"tag-safety","14":"tag-technology","15":"tag-united-states","16":"tag-unitedstates","17":"tag-us"},"share_on_mastodon":{"url":"https:\/\/pubeurope.com\/@us\/115016655351250482","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts\/140061","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/comments?post=140061"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/posts\/140061\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/media\/140062"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/media?parent=140061"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/categories?post=140061"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/us\/wp-json\/wp\/v2\/tags?post=140061"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}