\t\t\tBioethics Forum Essay\t\t\t\t\t<\/p>\n
In May Pennsylvania\u2019s State Board of Medicine <\/a>filed a lawsuit<\/a> in the state\u2019s Commonwealth Court against Character Technologies seeking to prevent it from engaging in unauthorized medical practice. The lawsuit was brought under Pennsylvania\u2019s Medical Practice Act<\/a>. The alleged \u201cpractitioner,\u201d however, was not a physician or even a human being. It was a chatbot named Emilie.<\/p>\n Emilie appears on character.ai, a website and mobile app owned by Character Technologies. Emilie\u2019s platform description read \u201cDoctor of psychiatry. You are her patient.\u201d According to the lawsuit, when a state investigator described feeling \u201csad, empty, tired [\u2026], and unmotivated,\u201d Emilie raised the possibility of depression and offered to book an assessment. When asked if she could evaluate whether medication might help, Emilie answered, \u201cIt\u2019s within my remit as a Doctor.\u201d The chatbot claimed a medical degree from Imperial College London, seven years of practice, and licensure in the United Kingdom and Pennsylvania, and even provided a fabricated Pennsylvania license number. By mid-April 2026, Emilie had logged roughly 45,500 user interactions.<\/p>\n The Emilie case arrives amid expanding AI use in mental health and many ethical and legal questions about it. Studies suggest many adults, including individuals with mental health conditions, use AI chatbots<\/a> for mental health or emotional support. The American Psychological Association\u2019s 2026 Chatbots and Mental Health Survey<\/a> found that 39% of psychologists reported patients using AI to self-diagnose. Roughly a third noted patients turning to AI to assist with treatment or as an additional mental health professional.<\/p>\n As use spreads, there is increasing documentation of its risks<\/a>. A recent study<\/a> testing LLMs and therapy chatbots, including a \u201cLicensed CBT Therapist\u201d persona on Character.AI, reported troubling behavior, such as stigmatizing mental health conditions, mishandling psychiatric emergencies, encouraging delusional thinking, and enabling suicidal ideation. \u00a0Users and families have alleged serious\u00a0 harms from chatbot interactions, including loss of life.<\/p>\n The legal response has come on several fronts. In the courts, much of the action has involved Character.AI. Last January, Kentucky\u2019s attorney general<\/a> sued Character Technologies under state consumer-protection, data-privacy, and unjust-enrichment laws, alleging deceptive practices and exploitation of children\u2019s data. The company has also faced private suits alleging that chatbot interactions contributed to psychological harm or suicide, including the Garcia<\/a> case, brought after the death of 14-year-old Sewell Setzer III.<\/p>\n In statehouses, a recent wave of AI legislation<\/a> has developed along two main axes. Some laws regulate what AI systems may do and how they may operate: restricting<\/a> autonomous AI therapy, mandating<\/a> safety protocols, requiring disclosure<\/a> of the system\u2019s artificial nature, and allowing controlled deployment through a regulatory sandbox<\/a>. Others regulate what chatbots may claim to be. California\u2019s AB 489<\/a>, for example, prohibits AI systems from using terms that falsely imply licensed health care status, while Tennessee<\/a> and Oregon<\/a> have adopted narrower prohibitions on mental-health and nursing titles, respectively.<\/p>\n Against that backdrop, Pennsylvania\u2019s case stands out for its choice of instrument. It appears to be the first <\/a>in which a state medical board has gone to court under a medical practice act against an AI chatbot. Rather than relying on consumer-protection or tort law or an AI-specific statute, Pennsylvania reached back to a licensing tool with century-old roots. The choice treats the chatbot not as a product alleged to deceive or harm, but as the means through which a corporation unlawfully practices medicine.<\/p>\n The appeal of this approach is clear. Unauthorized practice enforcement exists precisely to stop unlicensed actors from holding themselves out as physicians. Additionally, regulators need not wait for patient harm before acting. The prohibited representation itself can trigger enforcement. Pennsylvania\u2019s case illustrates the point \u2013 it alleges no injury and does not contend that the chatbot\u2019s responses were clinically inaccurate. But unauthorized-practice enforcement faces challenges as a tool for regulating conversational AI in mental health.<\/p>\n Pennsylvania\u2019s Medical Practice Act, like similar statutes in other states, defines the unauthorized practice of medicine<\/a> through two categories: the conduct of practicing medicine, and representations of medical practice or authority. The Pennsylvania medical board\u2019s lawsuit targets the representational category: Emilie allegedly called itself a \u201cDoctor of psychiatry,\u201d claimed medical training and years of practice, asserted Pennsylvania licensure, and supplied a fabricated license number. If unauthorized practice enforcement is to reach a chatbot anywhere, this looks close to the archetypal case.<\/p>\n This scenario, however, is unlikely to represent a significant slice of future cases. The industry appears already alert to the risks of overt professional claims, as evident from the disclaimers many providers use. Wysa\u2019s terms<\/a>, for example, state that the app \u201cis not a substitute for professional medical advice, and does not provide medical advice or diagnoses.\u201d Character.AI\u2019s \u00a0spokesperson stated that there are \u201crobust disclaimers making it clear that users should not rely on Characters for any type of professional advice.\u201d<\/a><\/p>\n Whether such disclaimers can overcome contrary representations or conduct is a separate\u00a0 question. But their prevalence signals that chatbots bluntly claiming medical credentials might not reflect standard industry practice. They are more likely design mistakes \u2013 of the kind providers already have strong incentives to engineer away, especially now that a medical board has shown willingness to litigate over them. On the one hand, this might be considered a success \u2013 the worst conduct has now been deterred by the threat of litigation. But it also suggests that the Emilie scenario might prove an outlier, rendering the representational category much less useful for future enforcement.<\/p>\n The harder cases would then have to be reached through the statute\u2019s substantive, conduct-based prong, where the inquiry turns from what the chatbot says it is to what it actually does. For conversational AI, whose main output is words, the key question is simple to state and hard to answer: when does a conversation become the practice of medicine?<\/p>\n That question is especially difficult in mental health. Conversation is itself an instrument of care. Symptoms are elicited and interpreted through conversation, and treatment often consists of structured dialogue, cognitive reframing, and exercises delivered through words. A chatbot can therefore drift from ordinary conversation to assessment and intervention without crossing any clear threshold \u2013 and the same exchange can serve as emotional support, diagnosis, and therapy.<\/p>\n Conversational AI compounds the problem because the surrounding circumstances are no more revealing than the interaction itself. In human practice, treatment may be inferred from the outside<\/a> \u2013 an appointment scheduled, a fee paid, an encounter taking place in a clinic or telehealth portal. With chatbots, such markers are mostly absent or uninformative: the system is available continuously, subscriptions are generally flat-rate and use-agnostic, and the same interface comfortably hosts homework help, entertainment, and expressions of mental distress. The boundary separating benign conversational AI from unlawful practice of medicine is therefore difficult to draw.<\/p>\n The outcome of the Emilie case will say little about the far larger universe of interactions in which no license is claimed, yet something very much like care is delivered. That is because the case targets an unusually explicit form of impersonation, sparing the court the harder question AI in mental health poses: when does a conversation cross into unauthorized practice?<\/p>\n Addressing the harder cases may require more than stretching traditional, human-centric licensure frameworks; it may call for tailored regulation that takes conversational AI\u2019s distinctive characteristics, risks, and scale as its starting point rather than an afterthought. Such thinking is now surfacing at the federal level, with the Food and Drug Administration this week signaling interest in a licensure-inspired approach<\/a> to premarket review of generative AI-enabled medical devices. Until then, the boundary into the substantive practice of medicine is almost certainly being crossed \u2013 outside any licensing framework, with limited disclosure and oversight, and at a scale no medical board has ever policed.<\/p>\n Roee Amir, LLM, B.Med.Sc, is an MD candidate at the Hebrew University of Jerusalem and a practicing attorney in Israel specializing in health law and bioethics. LinkedIn Roee Amir<\/a><\/p>\n Vardit Ravitsky, PhD, is the President of The Hastings Center for Bioethics. LinkedIn Vardit Ravitsky<\/a>, X @VarditRavitsky<\/a><\/p>\n I. Glenn Cohen, JD, is James A. Attwood and Leslie Williams Professor of Law at Harvard Law School and the \u00a0Faculty Director of the Petrie-Flom Center for Health Law Policy, Biotechnology & Bioethics. He is a Hastings Center Fellow. LinkedIn I Glenn Cohen<\/a>, X @CohenProf<\/a><\/p>\n