## The HEAL Framework: Assessing Health Equity in Machine Learning Performance
### Introduction
In today’s rapidly evolving technological landscape, machine learning (ML) and artificial intelligence (AI) promise significant advancements across various sectors, including healthcare. However, the transition from research to application in clinical settings necessitates an understanding of their impact on health equity. The Health Equity Assessment of Machine Learning Performance (HEAL) framework has been developed to ensure that ML-based health technologies function equitably.
### Understanding Health Equity and AI
Health equity refers to the fairness of opportunity for everyone to achieve their healthiest potential, recognizing that some populations may need more assistance due to systemic barriers. While AI fairness emphasizes equal performance across patient populations, health equity prioritizes improving outcomes for groups suffering from pre-existing health disparities. This difference is vital as technology should not only perform equally but also strive to reduce disparities in healthcare access and outcomes.
### The HEAL Framework
The HEAL framework aims to examine whether ML models perform adequately across different subpopulations, particularly those with the worst health outcomes. HEAL employs a structured methodology to analyze and enhance model performance concerning health equity. The framework involves comprehensive steps to identify disparities and assess model performance to avoid exacerbating existing inequities.
For instance, in a dermatology AI model, researchers used HEAL to evaluate the model’s effectiveness in diagnosing skin conditions. By incorporating demographic information and structured medical history, the model prioritized predicting conditions by aligning with the principle of health equity. This approach used the top-3 agreement metric, assessing how often the model’s predictions matched the most likely conditions identified by dermatologists.
### Case Study: Dermatology AI Model
A compelling application of the HEAL framework was demonstrated using a dermatology model trained to classify 288 skin conditions. Evaluated on a dataset comprising diverse cases from primary care providers in the USA and skin cancer clinics in Australia, the model’s performance was measured against pre-existing health outcomes using metrics like Years of Life Lost (YLLs) and Disability-Adjusted Life Years (DALYs). Sampling techniques ensured representation across race, sex, and age, providing a holistic evaluation of the model’s equity.
The model aimed to predict dermatologic conditions using patient photos and metadata. By comparing the top-3 agreement metric with health outcomes, researchers gauged if the model’s predictions aligned equitably across diverse subpopulations.
### Conclusion
The HEAL framework serves as a critical tool in evaluating the equity of ML-based healthcare technologies. By emphasizing the performance of models concerning health disparities, it ensures advancements in AI contribute positively to public health. However, future research is required to fully understand the causal steps in the care journey that impact the model’s efficacy in reducing real-world disparities.
### Call to Action
Start your 14 days trial with us and get access to our learning community. We build custom AI and Automations for businesses. Get in touch today and get your custom-built AI and Automations systems.
