As artificial intelligence becomes more embedded across healthcare delivery, questions around oversight, accountability, and governance are moving to the forefront.
AI-enabled care delivery models are reimagining healthcare for the modern era. New guidance from the Coalition for Health AI (CHAI) reflects growing expectations for structured governance across the AI lifecycle.
Health plan leaders now operate in an environment where AI systems influence clinical behavior before a claim is submitted.
AI is widely used in diagnosis support, risk stratification, treatment planning, and administrative decision-making. In recent years, due to a shift in how plan members seek care, AI has also become the initial point of care for many Americans. While healthcare is rapidly evolving, governance processes often lag behind implementation. As a result, coalition groups such as CHAI, have taken the lead in helping stakeholders determine the appropriate compliance and governance standards.
If inconsistent standards and fragmented responsibilities exist, AI-enabled care can contribute to payer risk across areas such as:
One of the overarching themes outlined in the CHAI guidance is that AI should support, not replace, clinical judgment. The framework notes that in high-stakes healthcare environments, a human should remain involved in all care decision-making. This is not simply a philosophical position but rather a risk-management requirement tied directly to accountability and patient safety.
Human clinical oversight also serves several payer-related functions, including establishing responsibility. The CHAI lifecycle requires organizations to define who will monitor performance, manage risk, and respond to adverse events. Without a clinician accountable for decisions, plans face unclear liability boundaries and difficulty defending coverage determinations.
Physician involvement also protects clinical appropriateness, as AI models can under-escalate or over-triage. Clinical review ensures that AI-powered advice aligns with a member’s complete medical picture instead of patient-provided inputs alone.
Oversight must also scale across populations, as many health plans manage large, heterogeneous memberships. The CHAI lifecycle, spanning definition through deployment and monitoring, creates repeatable governance processes that can be applied through a covered population. The framework also emphasizes safety, transparency, and security as operational principles for trustworthy health AI. For payers, physician-supervised AI is the approach most compatible with medical policy governance and defensible clinical decision oversight.
Members now routinely consult general-purpose AI tools for symptom interpretation and health advice before contacting a clinician or their health plan. Access is immediate, responses may appear personalized, and the interaction can appear clinical even when no clinical oversight exists. When access to care is delayed or inconvenient, individuals increasingly turn to search engines or consumer AI tools rather than a provider.
Many people no longer see these tools as simply experimental. According to a 2025 survey from the Annenberg Public Policy Center at the University of Pennsylvania, 63% of people who seek health information online consider AI-generated health information to be either “somewhat” or “very” reliable. When guidance appears credible, members may be more likely to act on it before entering the healthcare system.
As a result, the question for payers is no longer whether to adopt medical AI, but what standards must be non-negotiable to protect patient safety, clinical quality, and trust at scale. Consumer AI models operate without key clinical context. These tools do not incorporate plan structure, network availability, prior utilization history, or care management programs. Care protocols cannot be adjusted to adhere to predetermined medical policies or align recommendations with evidence-based pathways. As a result, medical advice could be inappropriate and lead to adverse outcomes.
The downstream effects of general-purpose LLMs vary. Some members defer care even when evaluation is necessary. Others interpret low-acuity symptoms as urgent and present to emergency departments or out-of-network settings unnecessarily. Both scenarios increase the total cost of care and introduce avoidable variation in utilization.
The CHAI health guidance requires transparency, monitoring, and defined accountability across the AI lifecycle. Systems influencing care should be auditable, and responsible parties should be identifiable. Consumer AI solutions are simply not equipped to meet these requirements–and disclaimers are simply not enough to truly provide high-quality, safe care.
CHAI offers operational expectations that payers can apply when evaluating AI-enabled vendors and care delivery partners.
It emphasizes that healthcare AI is intended to augment medical expertise and decision support rather than directly replace clinicians. For health plans, this translates into a practical procurement standard where defined physician oversight and feedback loops must exist to review medical AI that influences triage, diagnosis, or treatment decisions.
Governance must also be explicit and well-documented, as CHAI emphasizes that organizations should establish governance structures, standard operation procedures, and clearly defined responsibilities to monitor performance and manage risk. Plans should therefore expect defined escalation pathways, incident response processes, and monitoring metrics rather than general assurances of oversight.
Privacy and security must meet healthcare requirements, and accountability must remain identifiable. Trustworthy AI requires administrative, technical, and physical safeguards that protect confidentiality and system integrity. The guidance also calls for organizations to specify who is responsible for oversight, reporting, and corrective action across the AI lifecycle. In effect, the evaluation of AI vendors shifts from assessing technical performance alone to determining whether the technology can operate within a governed clinical and medical policy environment.
Counsel’s medical AI is designed to satisfy the highest compliance and governance standards for payers.
At the core of this system is continuous monitoring of both specificity and sensitivity, a critical requirement for safe triage at scale. By combining medical AI with physician oversight, Counsel asks the right questions, accurately assesses condition status, and escalates care appropriately. This architecture enables Counsel to deliver:
Additionally, Counsel’s platform leverages an agentic AI framework and retrieval-augmented generation (RAG) pipeline where each agent can be individually evaluated, constrained, or updated. Each member interaction is also monitored by independent safety agents to ensure every exchange meets clinical, legal, and payer-level standards for safety and compliance.
Counsel aligns with CHAI’s expectations by maintaining accountability, transparency, and clinical oversight. In doing so, we deliver scalable access to care while preserving quality, safety, and compliance.
Health plans preparing to advance their innovation strategy through the adoption of medical AI will need a partner whose technology can be governed as clinical infrastructure. Connect with Counsel to learn how you can reimagine healthcare for the modern era with responsible medical AI.
The Counsel Health editorial team is a multidisciplinary group of writers and editors dedicated to delivering clinically grounded, evidence-based health information. Their work is informed by real-world care delivery and guided by physician expertise, ensuring content is accurate, accessible, and trustworthy. By translating complex medical topics into clear, practical guidance, the team helps readers understand their health, explore care options, and make informed decisions in a rapidly evolving healthcare landscape.
Javier Monterrosa is a healthcare marketing leader who has spent his career driving growth across AI, metabolic health, interoperability, and EHR companies. He holds a Master’s in Analytics and has co-authored published research examining how strategic decisions shape business growth. Having grown up in Latin America, he is driven to partner with mission-driven teams committed to improving healthcare access and outcomes through responsible technology.
Our content is created for informational purposes and should not replace professional medical care. For personalized guidance, talk to a licensed physician. Learn more about our editorial standards and review process.
