In a 2025 study by Draelos et al., unsafe or potentially harmful responses occurred in up to approximately 43% of cases across several general-purpose LLMs. Yet, people frequently turn to consumer AI tools to make decisions about their health. According to Rock Health's 2025 Consumer Adoption Survey, 81% of individuals who use AI for health questions take a follow-on action, including:
Consumer AI tools are no longer passive sources of information. They are actively shaping clinical decisions outside of the supervision of health plans, introducing healthcare AI risk.
To provide a safe alternative, payers can deploy purpose-built medical AI that delivers immediate advice and treatment at the point of care. Responsible, AI-enabled models adopted as the modern front door can enable payers to shape care pathways in real time, aligning health decisions with clinical protocols, provider directories, networks, and existing ecosystems of health solutions.
According to a study published in KFF, people under the age of 30 turn to consumer AI tools due to cost (29%) and access (38%) barriers. As a result, health decisions have shifted to unmanaged general-purpose LLMs, with 42% of people in this same study noting not following up with a doctor or healthcare professional after consulting with these tools.
Even in the absence of these barriers, other studies have found increased health literacy from people. According to Gallup, 59% of people who use AI for health are using it to conduct research ahead of a doctor’s visit. Similarly, 56% of people use it to better understand in-person discussions after the appointment.
By not having clinical visibility into AI usage, payers continue experiencing high downstream utilization, out-of-network leakage, and higher total cost of care. Not only do consumer AI tools broaden the existing fragmentation in healthcare, but they also introduce clinical, utilization, and financial impacts for health plans.
Consumer AI tools were not built for clinical decision-making. Unlike purpose-built medical AI designed with agentic frameworks, they lack the foundational components required for effective risk management and safe care delivery:
Variability in quality of medical advice leads to two failure modes:
A 2024 Nature Communications study evaluating GPT-3.5 and GPT-4 in emergency department decision-making found that the models were overly cautious, generating more recommendations for admissions, imaging, and antibiotics than physicians reviewing the same cases. False-positive recommendations of this kind directly translate into avoidable utilization.
Inconsistent decision quality at the point of need creates unpredictable utilization patterns across the population. Health plans see the consequences in claims data, but they do not see the upstream conversations that drove the pattern.
These utilization patterns translate directly into financial and network impact:
Together, these patterns increase the total cost of care and erode network efficiency, despite significant investments in benefits design, navigation, and care management.
Expanding care access through the responsible use of AI in healthcare offers a new paradigm for payers.
Solutions like Counsel combine medical AI with physician oversight to deliver AI-enabled care through a secure messaging-based experience. To support large member populations at scale, Counsel embeds directly into a health plan’s existing member experience, removing the need to direct members to another application. Every interaction is secure and aligned to their clinical protocols, resulting in:
By shifting care decisions upstream, plans gain visibility into the interactions that drive utilization, without sacrificing the speed and convenience that draw members to consumer AI.
Counsel drives a 24% reduction in unnecessary emergency department visits compared with consumer-facing AI tools, reflecting more appropriate triage and clearer escalation pathways under physician supervision. At the same time, 96% of member concerns are resolved without escalation to downstream care, demonstrating the impact of context-aware clinical decision-making at the point of need.
Consumer AI adoption will continue to scale regardless of plan involvement. The risk is not adoption itself, but the absence of clinically governed intervention at the moment care decisions are made. Plans that establish an AI-enabled front door with responsible AI governance measures will be positioned to influence care pathways upstream, improving safety, utilization, and medical cost containment.
Request a demo to see how Counsel's AI-enabled care can drive value for your transformation strategy.
The Counsel Health editorial team is a multidisciplinary group of writers and editors dedicated to delivering clinically grounded, evidence-based health information. Their work is informed by real-world care delivery and guided by physician expertise, ensuring content is accurate, accessible, and trustworthy. By translating complex medical topics into clear, practical guidance, the team helps readers understand their health, explore care options, and make informed decisions in a rapidly evolving healthcare landscape.
Javier Monterrosa is a healthcare marketing leader who has spent his career driving growth across AI, metabolic health, interoperability, and EHR companies. He holds a Master’s in Analytics and has co-authored published research examining how strategic decisions shape business growth. Having grown up in Latin America, he is driven to partner with mission-driven teams committed to improving healthcare access and outcomes through responsible technology.
Our content is created for informational purposes and should not replace professional medical care. For personalized guidance, talk to a licensed physician. Learn more about our editorial standards and review process.
.png)
.png)
