Home / Key Decision Points / Assess quality of external AI product options
Guides

Assess quality of external AI product options

4 minutes Author: Danny Tobey and Keo Shaw

Why does this matter?

Preliminary quality checks can help protect against investing in a solution poorly suited to the target problem. AI products of interest may be marketed to appear suitable, but it is necessary to look past the sales pitch and really understand the product to make an informed decision. This diligence can prevent wasted time and effort by rooting out inappropriate AI product offerings and vendors at early stages of engagement.

  • It determines the product’s ability to address the target problem and how to measure the extent to which the problem is solved.
  • It ensures that the external AI product of interest and the associated developer/service provider meet the organization’s needs, confirm product performance claims, and limit exposure to legal, compliance, or ethical risks. 
  • It ensures that the organization is likely to capture value if the procurement opportunity is pursued.

How to do this?

    Step 1: Recruit a cross-functional team

    • Establish a cross-functional team with the breadth of skills and expertise necessary to perform the preliminary assessment. The team should include at least representation from the actual clinical users, statistical personnel with knowledge of validationThe process of establishing documented evidence demonstrating that a medical device, when used as intended, consistently produces the expected results. testing, IT or other personnel with knowledge of system alignment and integration feasibility, administrators with knowledge of clinical assets and workflows, and legal advisors. 
    • Bring in external advisors to assist with any aspects of the assessment that may not have in-house representation, such as a bias and ethics experts or community perspectives, where applicable.

    Step 2: Perform a preliminary quality assessment

    • Develop a quality assessment questionnaire to effectively communicate your organization’s information requests to vendors and to standardize the process.
    • Obtain relevant documentation through demonstrations and/or via question and answer sessions.
    • Ensure that information is provided to a satisfactory level of detail by the vendor’s relevant subject matter experts. It is very unlikely that one person will be able to provide all information sought.
    Assessment domainQualities to assess
    Quality and availability of the data used to train the external AI algorithm
    • Check if the data is diverse and representative to an extent that reflects the patient population within the proposed context of use.
    • Determine if the data used to train the model was collected under the same clinical circumstances and context that represents the intended useThe purpose for which a medical product, such as a drug or medical device, is intended according to the manufacturer's specifications or labeling. of the model. 
    • Confirm that the data is sufficiently conformant, complete, and plausible. 1,2
    • Determine if the data is appropriately labeled and whether it has been obtained legally and ethically.
    • Confirm that model testing was done using independent data from the data used to train the model, including considering independence of patients, sites, and time of data acquisition.
    • Confirm that any reference datasetsA Collection Of Structured Or Unstructured Data Used For Training Or Testing Machine Learning Models. are based on the best available methods to represent the clinical circumstances the model serves.
    Development and maintenance processes
    • Seek indicators of performance from academic literature or other healthcare delivery organizations who have adopted the product. 
    • If your organization has the expertise, data, and infrastructure, negotiate a ‘try-before-you-buy’ arrangement with the vendor to test the product in your organization’s digital infrastructure using local data.
    Technical compatibility
    • Check whether the product uses standard data exchange protocols.
    • Check whether the product can work with existing data formats, sources, and structures. 
    • Check whether the product is interoperable with existing software, such as electronic health record (EHR)Electronic Health Record is a digital record of a patient's medical history written by healthcare providers systems, imaging systems, or laboratory information systems.
    • Check whether the product can integrate within existing systems through application programming interfaces (APIs) or other integration methods. 
    • Check whether the product requires any specific hardware or infrastructure, such as high-performance computing (HPC) clusters, graphical processing units (GPUs) or specialized sensors.
    • Check whether the hardware used within the healthcare delivery context is compatible with the product’s requirements.
    Performance, accuracy, and reliability
    • Assess whether the product and vendor are capable of compliance with applicable laws and regulations. 
    • Map out proposed data flows and evaluate whether the product’s data collection, storage, and processing practices can meet the security and privacy requirements imposed by law and the organization.
    • For vendor’s supplying FDA regulated device software and/or storing data relating to a regulated device, determine whether the vendor has a robust understanding of the applicable regulations and whether it has the regulatory infrastructure to support the product (e.g. adequate quality system).
    Ethics and equity
    • Evaluate whether the product is designed to promote equity in healthcare delivery.
    • Assess whether there are baseline disparities in treatment and outcomes of the target condition within the context of use.
    • Investigate the vendor’s understanding of the biases their product may introduce to healthcare and the steps the vendor took in product development to mitigate biases. 
    • Examine whether the vendor has any self or third-party conducted impact assessments auditing for algorithmic bias.Algorithmic bias are the systematic errors or inaccuracies in computer algorithms that create unfair or unequal outcomes. . 
    • Understand the nature of the training dataA dataset from which a model is learned. A sample from a population of possible examples, and the statistical similarities of each class extracted, or more precisely the significant differences between classes are found. and model assumptions to evaluate the risk of bias in the population your organization serves. 
    • Determine whether the AI product’s design and implementation will align with your organization’s values and ethics.
    Legal and security
    • Assess whether the product and vendor are capable of compliance with applicable laws and regulations. 
    • Map out proposed data flows and evaluate whether the product’s data collection, storage and processing practices can meet the security and privacy requirements imposed by law and the organization.
    • For vendor’s supplying FDA regulated device software and/or storing data relating to a regulated device, determine whether the vendor has a robust understanding of the applicable regulations and whether it has the regulatory infrastructure to support the product (e.g. adequate quality system).

    Step 3: Decide whether or not to proceed to invest

    • Recommend whether or not to proceed to invest in a particular third party AI product. 

    “…Understanding the history of how people have been served in that space, the way in which healthcare or certain treatments or access to medicine has been stratified across different populations. In all of these cases, a deep understanding of the historical context is going to be what can inform questions about potential harms that could arise from choosing or building a tool.”

    Community Key Informant

    References

    1. Sendak, Mark, et al. “Development and validation of ML-DQA–a machine learning data quality assurance framework for healthcare.” Machine Learning for Healthcare Conference. PMLR, 2022.
    2. Corey, Kristin M., et al. “Assessing quality of surgical real-world data from an automated electronic health record pipeline.” Journal of the American College of Surgeons 230.3 (2020): 295-305.

    adopt health ai

    join us