AI Coalition Pushes Forward with Governance Plans

“AI is transforming the practice of medicine in ways that seemed unimaginable just two years ago,” CHAI co-founder John Halamka, MD, MS, president of the Mayo Clinic Platform and chair of the group’s new board of directors, said in a press release. “CHAI will bring together policymakers, technologists, healthcare providers, health plans, and a range of stakeholder advocates to develop guidelines and frameworks for evaluating AI. We have a shared mission to empower clinicians and patients with AI tools that maximize benefit and minimize harm.”

The partnership is the latest and largest effort to date to create guardrails around the fast-growing phenomenon of AI in healthcare. Health systems and hospitals are launching AI programs by the hundreds to address administrative and clinical pain points, but many worry that they’re going too fast too soon.

The Biden Administration has been pushing for a combined approach to AI governance, with an October 2023 Executive Order that puts the Health and Human Services Department and the Office of the National Coordinator for Health IT (ONC) at the forefront of regulatory efforts and a December 2023 final order that calls for more rules around transparency by the end of this year.

CHAI’s efforts have drawn support from US Food and Drug Administration (FDA) Commissioner Robert Califf, who spoke about AI at CES 2024 in January and addressed CHAI at a meeting earlier this week.

“As part of our AI strategy, the Agency is collaborating with public/private partners to develop a framework for assessing the potential risks and benefits of healthcare AI—this issue is too large to be contained within the FDA,” he told the group. “We’re also developing guidelines for the responsible deployment and ongoing monitoring of AI-driven health care solutions, including those using both adaptive and generative AI methods. The aim is to adapt general AI regulation and standards where needed to the unique characteristics of the health care sector. For instance, general AI regulations often stress the importance of accountability and transparency, which are also crucial in the health care domain due to the sensitive nature of health-related data.”

• First, the industry and federal regulators will need to monitor AI tools continuously, since the technology is constantly evolving and the threat of drift over time. “With the proliferation of AI applications and the fact that they evolve over time, it is unclear how the performance of the models will be monitored at the scale that will be needed,” he noted.
• Second, AI governance must include complete follow-up of the population affected by the AI tool. “The lack of an interoperable national approach to enabling follow-up of patients leads to a situation in which none of our health systems have a systematic ability to do the requisite monitoring of the model except for the duration of an acute care hospital admission,” Califf pointed out.

Califf also sounded a note of caution on “effectiveness” metrics for AI evaluation, saying he worried those analyses might focus on a health system’s financial interests rather than whether the technology can improve clinical outcomes.