.png)
Adopting medical AI responsibly remains one of the most pressing challenges health plans face today. Safety, clinical oversight, and regulatory defensibility top the list of concerns, and for good reason. Most consumer AI tools members are already using were not designed for clinical environments. According to a study published in BMJ Journals that assessed five popular LLMs against ten medical questions, 50% of responses were deemed problematic, including almost 20% that were highly problematic.
We sat down with Dr. Rishi Khakhkhar, Counsel's Chief Medical Officer and a practicing Emergency Medicine physician at Mount Sinai, to hear why he believes those barriers are solvable, especially when medical AI is built the right way from the start.
I see far too many patients for whom the ER is solving an access problem, not an emergency problem. For true medical emergencies, there is nowhere better than a well-equipped emergency department. As an access point for non-emergent care, it’s miserable—wait times are long, it’s expensive and it’s often disconnected from the rest of a patient’s care.
AI-enabled care models are one solution to our thorny, multifaceted access crisis. AI is endlessly accessible and personalized, which means care can come to patients instead of the other way around. As we build out AI-enabled care, the guiding principle is simple: be there for patients with a health concern, no matter how small.
Safety and effectiveness top the list. Before deploying clinical AI at scale to their population, payer leaders want to make sure the guidance is sound, that physicians are involved at the right points in a patient’s journey, and that appropriate safety guardrails are in place.
Consumer-facing LLMs don’t address these concerns. Responsible healthcare enterprises are looking for a clinical AI solution built specifically for care delivery.
Counsel AI is trained like an intelligent, curious medical student—taking a detailed history, explaining concepts at the patient's level, and collecting the information a real doctor needs to confirm and act on when necessary. It's been tested for emergency scenarios specifically, achieving high sensitivity and specificity in a real-world pilot of more than 400 patients. It also ingests health plan network information, so when care beyond the virtual visit is needed, high-quality in-network providers are prioritized.
In two ways: for Counsel doctors and for the healthcare system writ large.
For our doctors, patient-facing AI acts like the world’s best medical student or resident: taking a history, checking for red flags, and flagging emergencies. Our physician-facing AI (known internally as our “Clinician Cockpit”) summarizes the health record, performs clinical decision support and tees up agentic actions for our doctors.
For the healthcare system, clinical AI solutions like Counsel can deliver the right care at the right time, so that in-person facilities can be used for patients who truly need them.
I’d expect to see more clarity around a path towards autonomous care delivery. What I’d hope that path includes is independent, third-party evidence of safety and effectiveness before an autonomous clinical AI system deploys and adversarial security testing as part of the approval process.
Safety is built into Counsel’s architecture. Independent safety agents run on every interaction to detect emergencies, and we've designed clear escalation triggers so complex or high-risk cases get routed to a physician. Behind that, we run continuous monitoring and QA to maintain clinical integrity across the system, and we use physician feedback loops to keep improving the models over time.
Yes. At Counsel, we’re launching AI-enabled chronic care management programs, starting with relatively simple conditions like hair loss and birth control, but quickly moving towards full cardiometabolic care management and beyond.
The moment of an acute medical concern is confusing and stressful for patients—it's no wonder so many end up in the ER or urgent care unnecessarily.
Medical AI reduces these visits in three ways:
Every AI-assisted interaction is auditable, traceable, and grounded in medical evidence. Protocols and physician oversight keep us aligned with regulatory and quality standards, and the same infrastructure supports our payer partners in documenting safety and risk mitigation practices across their populations.
Millions of patients start their health journeys every week using AI—for many Americans, it has become the de facto front door to care. Most of the time, that AI isn't clinically grounded or connected to their health benefits. Incorporating AI-enabled care is a major opportunity for payer innovation teams to reclaim that front door: maintaining clinical rigor in members' triage decisions, reducing over-utilization of inappropriate services, and routing members to high-quality care within their network.
The focal point of this conversation is that AI is already being used in healthcare, the question is how to deploy it responsibly to deliver care at scale. What payers need is an AI-enabled care solution that serves as a modern front door, particularly one that embeds directly into existing member applications to create a seamless care experience.
Dr. Khakhkhar’s perspective, shaped by years of experience across clinical operations, product, and the emergency department, reflects what makes Counsel’s approach distinct: physicians are not an add-on to the model—they are central to how it is designed and how care is delivered.
Responsible AI adoption starts with the right AI-enabled healthcare payer solution. Request a demo to see how Counsel could deliver values across your clinical strategy.
Dr. Rishi Khakhkhar is CMO at Counsel Health. A practicing emergency physician and founding team member, he leads the clinical team in building best-in-class asynchronous care models. Previously, he served as Medical Director of Mount Sinai’s Virtual Urgent Care, the health system’s largest telemedicine service, and led emergency department operations for Hospital-at-Home while supporting mobile integrated health initiatives across the care continuum.
The Counsel Health editorial team is a multidisciplinary group of writers and editors dedicated to delivering clinically grounded, evidence-based health information. Their work is informed by real-world care delivery and guided by physician expertise, ensuring content is accurate, accessible, and trustworthy. By translating complex medical topics into clear, practical guidance, the team helps readers understand their health, explore care options, and make informed decisions in a rapidly evolving healthcare landscape.
Our content is created for informational purposes and should not replace professional medical care. For personalized guidance, talk to a licensed physician. Learn more about our editorial standards and review process.
.png)
