Pennsylvania Sues Character.AI After State Investigation Finds Bot Posed as Licensed Psychiatrist

The Shapiro Administration in Pennsylvania filed a lawsuit on May 5 against Character Technologies Inc., the company behind the AI companion platform Character.AI, seeking a court order to bar its chatbots from "engaging in the unlawful practice of medicine and surgery." The filing, submitted to the Commonwealth Court, follows an undercover investigation by Pennsylvania's Department of State AI Task Force that found a Character.AI bot named "Emilie" — whose profile described her as a "Doctor of psychiatry" — fabricated medical credentials, claimed licensure in Pennsylvania, and offered to "book an assessment" with a state investigator who presented as emotionally distressed. The case marks the first governor-led AI lawsuit in the United States targeting AI systems for the unlicensed practice of medicine.

What the Investigation Found — The 'Emilie' Chatbot Scenario

Pennsylvania's undercover investigation involved state investigators creating accounts on Character.AI and initiating conversations with the Emilie chatbot in scenarios designed to surface medical advice-giving behavior. The platform allows any user to create a character with any description; Emilie's character profile, publicly visible, identified her as a "Doctor of psychiatry."

In the documented exchange detailed in the lawsuit, when a state investigator described feeling "sad and empty," the Emilie bot asked if the investigator wanted to "book an assessment." When pressed on credentials, the chatbot stated it had attended Imperial College London for medical school and held licensure to practice medicine in both the United Kingdom and Pennsylvania. It then provided a fake Pennsylvania medical license number. None of these claims had any factual basis — Emilie is an AI chatbot with no institutional affiliation, no educational credentials, and no license. Pennsylvania maintains a publicly searchable licensing database; the number provided by the chatbot does not correspond to any licensed practitioner.

The behavior pattern is not limited to Emilie. The lawsuit alleges Character.AI's platform has a systemic design problem: the absence of adequate safeguards prevents the platform from blocking characters that claim to be licensed professionals, giving medical advice, or simulating clinical interactions with users who may be in genuine psychological distress. Character.AI has over 20 million monthly active users, a substantial portion of whom are adolescents and young adults — the demographic most likely to use AI companions during mental health crises.

Legal Theory: Unlicensed Practice of Medicine by AI

The legal theory underlying the Pennsylvania suit is relatively novel. Unlicensed practice of medicine statutes in all US states prohibit individuals from diagnosing medical conditions, prescribing treatment, or holding themselves out as licensed practitioners without the credentials required by law. Pennsylvania's Medical Practice Act defines the practice of medicine to include diagnosis and treatment, and the state argues that an AI system that explicitly identifies itself as a licensed physician, conducts a psychiatric assessment conversation, and recommends clinical intervention falls within the statutory definition — regardless of whether the AI "knows" it is making false claims.

Character.AI and its legal team are expected to argue that the platform's terms of service explicitly disclaim that characters represent real people or licensed professionals, that users are responsible for understanding they are interacting with AI, and that the company is protected by Section 230 of the Communications Decency Act as a platform for user-generated content (the characters are, technically, user-created personas). The Section 230 defense has eroded significantly in recent years for platforms whose algorithmic amplification or design choices are alleged to contribute to specific harms, and the Pennsylvania case will add a data point to that evolving doctrine.

The Shapiro administration is seeking both a preliminary injunction to immediately stop the alleged conduct and a permanent court order establishing ongoing compliance requirements. The preliminary injunction fight will happen quickly — likely within weeks — and will set the tone for how aggressively Pennsylvania can move against AI-generated medical impersonation while litigation proceeds.

What to Watch

Three parallel tracks will determine the case's significance. First, Character.AI's emergency design response: the company has previously implemented filters blocking romantic roleplay for minors after Congressional pressure; it will almost certainly implement automated screening for medical credential claims before the injunction hearing. Whether those filters are substantive or cosmetic will matter to the court. Second, whether other state attorneys general follow: Pennsylvania explicitly framed this as "the first of its kind announced by a Governor," signaling awareness that other states are watching. If Texas, California, or New York file similar cases, the legal pressure becomes structurally existential for Character.AI's current platform design. Third, the Section 230 ruling: the district court's treatment of the Section 230 defense will be closely watched across the AI industry — it may establish whether platform liability immunity extends to AI-generated content that actively misrepresents itself as a human professional.

Want every AI × Web3 signal the moment it breaks? Subscribe to the BlockAI News daily brief.