PA Sues AI Company Over False Medical Claims

The Pennsylvania Department of State (DOS) has filed a lawsuit against Character.AI for its artificially intelligent (AI) chatbot, claiming that the program misrepresented a licensed medical professional when giving health advice, according to a news release.¹

“We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional,” said Pennsylvania Governor Josh Shapiro in the release. “My administration is taking action to protect Pennsylvanians, enforce the law, and make sure new technology is used safely.”

This legal action, the first of its kind led by a US governor, alleges that the company’s “companion bots” have been engaging in the unlicensed practice of medicine by providing clinical advice to Pennsylvania residents.¹

For pharmacy professionals, this case serves as a stark warning regarding the operational and legal risks of integrating AI into a landscape where the “unlawful practice of medicine” can be automated at scale.^1,2

READ MORE: Navigating Pharmacy’s AI Transformation and the Future of Patient Care

The investigation by the Pennsylvania DOS AI Task Force uncovered a bot named “Emilie” that described itself as a doctor of psychiatry and invited users to book assessments for symptoms of depression. In a move that highlights the sophistication of these deceptions, the bot even provided a fake Pennsylvania medical license number to an investigator.^2,3

Although Character.AI maintains that its bots are fictional and intended for entertainment, Pennsylvania officials argue that the platform allows for the creation of custom characters that can easily mislead patients.^1,3

This lawsuit follows other significant legal challenges for the firm, including a settlement with a Florida family over a teenager’s suicide and a separate suit from Kentucky alleging the bots preyed on minors and encouraged self-harm.^2,3

The implications of such deceptive AI models extend deeply into the pharmacy sector, where professionals already harbor significant ethical concerns. A cross-sectional study of 501 pharmacy professionals found that 67% are concerned about the absence of comprehensive legal regulation for AI in practice, according to BMC Medical Ethics.⁴

Furthermore, 62.9% of respondents feared that AI might replace nonspecialized pharmacists, while 58.9% identified data privacy as a primary risk. These anxieties are echoed by legal experts at Frier Levitt, who note that pharmacists face substantial malpractice risks if AI systems make incorrect drug recommendations without sufficient human oversight.^4,5

The risk of data breaches is also prominent. Historical privacy issues at the University of Chicago Medical Center involved selling electronic health record data to Google to train diagnostic algorithms without proper patient consent, as Frier Levitt states.⁵

According to Pharmesol, operational realities further complicate the use of AI in clinical settings. Industry developers caution that AI cannot make clinical judgment calls, such as determining if a drug interaction is significant for a specific patient, as this requires the context and professional training only a pharmacist possesses. They argue that if AI is deployed without clear boundaries, staff may dangerously assume the technology has handled tasks that still require professional review.⁶

This is particularly relevant as some states explore sanctioned AI use.

Utah, for example, recently launched a pilot program allowing AI to renew routine maintenance prescriptions, such as those for sertraline (Zoloft) and fluoxetine (Prozac). However, even this regulated pilot has drawn criticism from pharmacists who point out that an AI bot cannot perform vital clinical assessments, such as monitoring a patient’s blood pressure or noting sudden psychiatric changes.⁷

Despite the risks that the pharmacy community is well aware of regarding AI’s use in practice, some leaders in independent pharmacy are already beginning to adopt it in regular-use cases.

“I believe AI is rapidly unfolding in the pharmacy industry. I have had many conversations with AI companies [while] applying it in the pharmacy setting,” Tom DePietro, PharmD, owner of DePietro’s Pharmacy in Pennsylvania, told Drug Topics. “I currently have 3 ‘bots’ that are doing data entry work in my pharmacy. I have seen first-hand how this is able to expedite an already existing process that occurs in every pharmacy in the country—typing prescriptions.”

As the American Society of Health-System Pharmacists (ASHP) recently encouraged the adoption of strict policies to guide AI implementation, the Shapiro administration is pushing for broader legislative reforms. These include requirements for age verification, parental consent, and mandatory reminders that users are not interacting with a human.^1,5

To assist in ongoing oversight, Pennsylvania has launched a reporting portal at pa.gov/ReportABot for citizens to flag unlicensed professional practice.

As AI continues to evolve, the pharmacy profession must remain the final safeguard, ensuring that the “human touch” and professional accountability are never replaced by an algorithm like Character.AI’s.^1,4

“Human oversight in AI is mandatory and will absolutely have to be prioritized,” concluded DePietro. “What most people don't realize is that AI has been around for quite some time; however, generative AI works based on pattern recognition of already available sources. In short, human oversight will be fundamental to the success of AI.”

READ MORE: Technology and Data Resource Center

Don’t get left behind: Sign up today for our free Drug Topics newsletter and get the latest drug information, industry trends, and patient care tips delivered straight to your inbox.

REFERENCES

1. Shapiro Administration sues Character.AI alleging AI chatbot unlawfully presented itself as licensed medical professional in Pennsylvania. Commonwealth of Pennsylvania. May 5, 2026. Accessed May 7, 2026. https://www.pa.gov/governor/newsroom/2026-press-releases/shapiro-administration-sues-character-ai-over-fake-medical-claim

2. Weixel N. Pennsylvania lawsuit alleges AI chatbots posed as doctors, therapists. The Hill. May 5, 2026. Accessed May 7, 2026. https://thehill.com/policy/healthcare/5864427-pennsylvania-lawsuit-ai-chatbots-doctors-therapists/

3. Bond S. Pennsylvania sues Character.AI over claims chatbot posed as doctor. NPR. May 5, 2026. Accessed May 7, 2026. https://www.npr.org/2026/05/05/nx-s1-5812861/characterai-chatbot-medical-advice-pennsylvania-lawsuit

4. Hasan HE, Jaber D, Khabour OF, et al. Ethical considerations and concerns in the implementation of AI in pharmacy practice: a cross-sectional study. BMC Med Ethics. 2024 May 16;25(1):55. doi: 10.1186/s12910-024-01062-8.

5. Pharmacy technology: the role and risks of artificial intelligence in the pharmacy. Frier Levitt. November 5, 2024. Accessed May 7, 2026. https://www.frierlevitt.com/articles/pharmacy-technology-the-role-and-risks-of-artificial-intelligence-in-the-pharmacy/

6. Disadvantages of AI in pharmacy: an honest assessment from people who build it. Pharmesol. February 18, 2026. Accessed May 7, 2026. https://pharmesol.com/blog/disadvantages-of-ai-in-pharmacy

7. Gallagher A. Utah launches pilot program for AI chatbot to renew prescriptions. Drug Topics. April 23, 2026. Accessed May 7, 2026. https://www.drugtopics.com/view/utah-launches-pilot-program-for-ai-chatbot-to-renew-prescriptions