Assess quality of external AI product options
Why does this matter?
Preliminary quality checks can help protect against investing in a solution poorly suited to the target problem. AI products of interest may be marketed to appear suitable, but it is necessary to look past the sales pitch and really understand the product to make an informed decision. This diligence can prevent wasted time and effort by rooting out inappropriate AI product offerings and vendors at early stages of engagement.
- It determines the product’s ability to address the target problem and how to measure the extent to which the problem is solved.
- It ensures that the external AI product of interest and the associated developer/service provider meet the organization’s needs, confirm product performance claims, and limit exposure to legal, compliance, or ethical risks.
- It ensures that the organization is likely to capture value if the procurement opportunity is pursued.
How to do this?
Step 1: Understand which regulatory schemes apply to the AI product
- Examine whether the planned use of the AI product is regulated by a health authority (e.g.,US FDAFood And Drug Administration Is A Government Agency That Regulates The Safety And Effectiveness Of Food, Drugs, Medical Devices, And Other Products.) or is subject to health privacy laws (e.g., HIPAAHealth Insurance Portability And Accountability Act, A Us Law That Regulates The Handling Of Sensitive Medical Information.).
- If the product is regulated, account for specific quality requirements during the preliminary assessment.
Step 2: Recruit a cross-functional team
- Establish a cross-functional team with the breadth of skills and expertise necessary to perform the preliminary assessment. The team should include at least representation from the actual clinical users, statistical personnel with knowledge of validationThe process of establishing documented evidence demonstrating that a medical device, when used as intended, consistently produces the expected results. testing, IT or other personnel with knowledge of system alignment and integration feasibility, administrators with knowledge of clinical assets and workflows, and legal advisors.
- Bring in external advisors to assist with any aspects of the assessment that may not have in-house representation, such as a bias and ethics experts or community perspectives, where applicable.
Step 3: Perform a preliminary quality assessment
- Develop a quality assessment questionnaire to effectively communicate your organization’s information requests to vendors and to standardize the process.
- Obtain relevant documentation through demonstrations and/or via question and answer sessions.
- Ensure that information is provided to a satisfactory level of detail by the vendor’s relevant subject matter experts. It is very unlikely that one person will be able to provide all information sought.
|Assessment domain||Qualities to assess|
|Quality and availability of the data used to train the external AI algorithm|
|Development and maintenance processes|
|Performance, accuracy, and reliability|
|Ethics and equity|
|Legal and security|
Step 4: Decide whether or not to proceed to invest
- Recommend whether or not to proceed to invest in a particular third party AI product.
“…Understanding the history of how people have been served in that space, the way in which healthcare or certain treatments or access to medicine has been stratified across different populations. In all of these cases, a deep understanding of the historical context is going to be what can inform questions about potential harms that could arise from choosing or building a tool.”Community Key Informant
- Sendak, Mark, et al. “Development and validation of ML-DQA–a machine learning data quality assurance framework for healthcare.” Machine Learning for Healthcare Conference. PMLR, 2022.
- Corey, Kristin M., et al. “Assessing quality of surgical real-world data from an automated electronic health record pipeline.” Journal of the American College of Surgeons 230.3 (2020): 295-305.