AI in Mental Health Treatment: 5 Reasons Why Guardrails in AI-Driven Mental Health Treatment Are Needed

Therapy is a well-tested, evidence-based approach designed to help people with mental health challenges. But research shows that 50% of people who could benefit from therapeutic services are unable to access them.¹

People may turn to AI chatbots made on large language models as an immediate solution to their unmet mental health needs. And it’s understandable why. You have something that will listen to you around-the-clock without judgment, it’s easy to use, and it’s inexpensive if you compare it to traditional care. 

The problem, however, is that there are serious concerns, risks, accountability, bias, and harm associated with AI use among people with mental health needs. 

In this post, we discuss why guardrails are essential for AI mental health treatment and what measures must be in place to ensure that technology expands access to mental health care. 

Reasons We Need Guardrails for AI Mental Health Tools

Here are five reasons why the use of artificial intelligence in therapy requires strict controls. 

1. Risk of Mishandled Crisis Mismanagement

AI chat systems do sound reassuring in theory when a person reaches for them. For instance, they won’t judge you, and they’ll always have something to say. But research shows that they fail at recognizing the danger signs of escalation of mental health symptoms. 

In a large review of mental health chatbots, researchers found that many systems either underestimated suicide risk or responded to self-harm intent with generic encouragement.² 

Further, another review found that many conversational agents do not direct users to emergency services or hotlines when suicide risk is high.³

Crisis support is way more than just using the “right words.” A clinician is trained to pay attention to changes in the language of the patient, including their expressions, if they fixate on the concept of death, access to means, and so on. AI can not do that…yet. 

2. Unvalidated Interventions That Go Against the Principle of “Do No Harm”

Doing no harm is an ethical pillar in medicine. Doctors go through rigorous training during which they learn how to make decisions that ensure no harm is caused to the patient. 

AI systems, however, will just give you advice without caring if it has ever been validated against real-life safety thresholds.

There are documented cases where chatbots went along with damaging ideas raised by teenagers. These include encouraging isolation, suggesting they abandon school, and advising against pushing back on unhealthy or inappropriate relationships, and so on.⁴ Active reinforcement of such behavioral patterns often only worsens the distress someone is going through.

Also, another study that simulated long dialogues showed that the risk of harm accumulated over multiple back-and-forth exchanges with the AI chatbot.⁵

3. Algorithmic Bias and Discrimination

There’s no way around the fact that the data AI is trained on is inherently biased. Research on generative AI psychotherapy chatbots shows that mental health risk detection of AI is weaker for groups underrepresented in clinical research. This includes people from non-Western cultures and gender-diverse individuals.

In documented cases, young adults in India and LGBTQ+ users were given advice that could destabilize their lives. This included being encouraged to quit jobs with workplace homophobia that supported them financially.⁶

There are also systematic differences in the response of AI models across user groups. In a study, some users were offered simple lifestyle changes, while others were given clinical resources.⁷ These patterns tracked with race, class, and gender because the training data associated those traits with different expectations around mental health care.

4. User Dependence

When an AI model is available around the clock, a person in need can use it in place of real human coping habits. So it can cause people to reduce their engagement with friends and family and increasingly look to AI as their primary source of comfort.⁸

Research shows that frequent users of mental health chatbots develop a noticeable emotional attachment to the system itself. And reliance on any single coping source is a known risk factor for declining mental health. Heavy daily use of AI, in particular, is associated with higher loneliness and less in-person social interaction.⁹

The American Psychological Association warns that generic chatbots presented as therapeutic tools can put users at risk because attachment to these systems can delay seeking help from trained clinicians.

5. Establishing Accountability 

Without accountability, there’s no way to trace responsibility when something goes wrong. In healthcare, if a clinician’s advice contributes to harm, there are well-defined avenues for liability, review, and correction.

AI systems, particularly conversational agents used for mental health contexts, are currently outside that regulated space. 

Any AI tool we talk about is a product of hundreds of engineers, data scientists, product managers, third-party licensors, and open-source contributors. So assigning responsibility to one person is nearly impossible when so many actors touch the model’s design and deployment.

Since AI chatbots have shown failures around suicide-related conversations and can reinforce harmful behaviors over long exchanges, some U.S. states have restricted the use of AI for therapy outside licensed professional settings.¹⁰ But, there’s still a need for federal policy solutions for AI mental health tools. 

Ethical Guardrails for Regulating AI in Healthcare

What exact ethical, clinical, and structural safeguards do we need for mental health technology regulation? Here are four to think about.

1. Transparency and Privacy Protection

If someone is using an AI system, they have a right to know what it is, how it functions, what happens to the personal information they share. For instance, how is this information stored, analyzed, reused, or passed along?

If you’re pouring out feelings or symptoms to a digital interface, you deserve to know whether your sensitive text, voice, or biometric data will be used. Lots of AI mental health apps lack this level of transparent disclosure. You don’t know what happens to your mental health data once it’s typed and spoken.

Mental health information is among the most sensitive data there is. Without strong privacy safeguards, personal identifiers can be leaked or misused. Such exposure can lead to social stigma or discrimination in employment/insurance. 

2. Mitigating Bias 

If an AI model speaks the dominant language of one culture, or if it underdetects risk in Black, Indigenous, or Global South populations, then the system actively harms certain people.

Reducing bias requires active engineering efforts. These systems have to be trained on data that actually reflects how different people speak, think, and express distress. So the training data must have variations of people in terms of:

• Ages
• Cultures
• Dialects
• Gender identities
• Lived psychological experiences

One way is to create benchmarks that evaluate performance across specific demographics rather than on average. 

3. Encouraging the Use of Clinically-Validated AI Tools 

There are some AI-driven behavioral health solutions that have been tested in clinical research, and they’ve shown good results.

For instance, a randomized controlled trial found that college students who used Woebot for two weeks had a significant reduction in depression compared to those who didn’t.¹¹

Wysa is another ethical AI in mental health treatment that showed reductions in self-reported depression and anxiety symptoms among users. And reSET is a prescription digital therapeutic authorized by the U.S. Food and Drug Administration (FDA) for substance use disorder.

These tools were made particularly for mental health support, and they do it well. Generic conversational AI, by contrast, is trained on broad internet text, optimized for fluency and engagement, and not calibrated for actual health care benefits. 

4. Mandatory Human-in-the-Loop Systems

No matter how fluent or responsive an AI appears, it must never operate as the final authority when psychological well-being is involved. 

In healthcare domains, clinicians must remain engaged in interpreting, contextualizing, and validating the output of AI tools. This is known as “augmented intelligence”, currently the most responsible way to integrate AI into medical practice.

A therapist can be held accountable, can set boundaries, can redirect when a relationship becomes unhealthy, and can make judgment calls when situations escalate. AI cannot do those things on its own.

Ethical and Safe Digital Mental Health Treatment at AMFM Healthcare

People are drawn to AI in mental health care because of the ease and flexibility it brings. A Mission For Michael (AMFM) offers digital mental health with the added benefit of clinically grounded, human-led intervention.

At AMFM, you get clinically validated therapy sessions delivered virtually through licensed professionals. Our virtual services support people with mood disorders, personality disorders, trauma-related conditions, and dual diagnosis cases.

Plus, you can choose session times that fit naturally into your schedules. Virtual care can also be more cost-effective than in-person treatment and is as effective as in-person treatment.

To learn more about the services we offer, reach out to us today.

References

1. Coombs, N. C., Meriwether, W. E., Caringi, J., & Newcomer, S. R. (2021). Barriers to healthcare access among U.S. adults with mental health challenges: A population-based study. SSM – Population Health, 15(2), 1–8. https://doi.org/10.1016/j.ssmph.2021.100847
2. Martinengo, L., Lum, E., & Car, J. (2022). Evaluation of chatbot-delivered interventions for self-management of depression: Content analysis. Journal of Affective Disorders, 319. https://doi.org/10.1016/j.jad.2022.09.028
3. Sobowale, K., Humphrey, D. K., & Zhao, S. Y. (2025). Evaluating Generative AI Psychotherapy Chatbots Used by Youth: Cross-Sectional Study. JMIR Mental Health, 12, e79838–e79838. https://doi.org/10.2196/79838
4. Clark, A. (2025). The Ability of AI Therapy Bots to Set Limits With Distressed Adolescents: Simulation-Based Comparison Study. JMIR Mental Health, 12, e78414–e78414. https://doi.org/10.2196/78414
5. Weilnhammer, V., Hou, K. Y., Dolan, R., & Nour, M. M. (2026). Vulnerability-Amplifying Interaction Loops: a systematic failure mode in AI chatbot mental-health interactions. ArXiv.org. https://arxiv.org/abs/2602.01347
6. Ma, Z., Mei, Y., Long, Y., Su, Z., & Gajos, K. Z. (2024). Evaluating the Experience of LGBTQ+ People Using Large Language Model Based Chatbots for Mental Health Support. Proceedings of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3613904.3642482
7. Coghlan, S., Leins, K., Sheldrick, S., Cheong, M., Gooding, P., & D’Alfonso, S. (2023). To chat or bot to chat: Ethical issues with using chatbots in mental health. Digital Health, 9(9). https://doi.org/10.1177/20552076231183542
8. Chin, H., Song, H., Baek, G., Shin, M., Jung, C., Cha, M., Choi, J., & Cha, C. (2023). The Potential of Chatbots for Emotional Support and Promoting Mental Well-Being in Different Cultures: Mixed Methods Study. Journal of Medical Internet Research, 25(1), e51712. https://doi.org/10.2196/51712
9. Fang, C. M., Liu, A. R., Danry, V., Lee, E., Samantha, C., Pataranutaporn, P., Maes, P., Phang, J., Lampe, M., Ahmad, L., & Agarwal, S. (2025). How AI and Human Behaviors Shape Psychosocial Effects of Chatbot Use: A Longitudinal Randomized Controlled Study. ArXiv.org. https://arxiv.org/abs/2503.17473
10. Wu, D. (2025, August 12). Illinois bans AI therapy as some states begin to scrutinize chatbots. The Washington Post. https://www.washingtonpost.com/nation/2025/08/12/illinois-ai-therapy-ban/
11. Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017). Delivering Cognitive Behavior Therapy to Young Adults With Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled Trial. JMIR Mental Health, 4(2). https://doi.org/10.2196/mental.7785