Evaluate expansion to new settings

Why does this matter?

Because each health care setting is unique, the AI product that one setting designs and integrates cannot be seamlessly transitioned for immediate use in another setting. Expanding the use of a medical AI product to other settings includes any change in the AI product’s scope of use, including integration in a new location, use by a different group of users or organization or use for a broader purpose. It is critical to evaluate an AI product within each local context of use (see how to generate evidence of safety, efficacy, and equity).

• Healthcare organizations may need to adapt the AI product by retraining the model on new data, fine-tuning its parameters, or making other adjustments to ensure safe, effective and equitable performance in the expanded setting.
• Responsible expansion of an AI product requires significant resource investment from cross-functional stakeholders to fully understand the expanded use case and evaluate key decision points in that new context. All elements of AI product design and integration should be reconsidered in light of the expanded setting so that a healthcare organization can make informed decisions about expansion.

How to do this?

Step 1: Ensure clear understanding of the AI product in the current setting

• Document key metrics of an AI product (e.g. what an AI product measures, improves, and requires in terms of capital and technical resources) to get an overview of how the product is currently working. 
• Analyze AI product’s performance in the current setting to predict how the AI product may work in an expanded setting and flag potential challenges (see how to monitor the AI product under the lifecycle management phase).

“How do we take expertise that exists through the [business unit] and start to scale the technology and introduce it into a slightly different care or diagnosis area…. so that we can leverage the same underlying infrastructure and techniques. That’s, what I would call our next frontier.”

Technical leader

Step 2: Identify and understand the potential expanded setting

• Evaluate potential expanded settings to document the proposed use and goals of the AI product, existing technological infrastructure, capital available, needs and prospects for integration. 
• Use this information to determine if the setting is a good candidate for expansion of the AI product.
  • Much of this work will be similar to the definition and assessment work described in guides under ‘Define AI product specification’ and ‘Develop success measures.’ However, when expanding to an expanded setting, health systems may need to coordinate with external partners to acquire all of the necessary information.

Step 3: Consider regulatory requirements in the expanded setting

• Reassess the regulatory requirements applicable to the AI product in the expanded setting.
• Changes in regulatory consideration can have significant impact on project timing and resource requirements.

Step 4: Assess the feasibility and scalability of expansion

• Use the information collected in steps 1 – 3 to assess the feasibility and scalability of expanding the use of the AI product.
• This includes evaluating the availability of appropriate resources, such as infrastructure, personnel and funding, needed to support the expansion.

“There’s a very long history of rolling things out from a local hospital to a regional program. Most of my work is regional, meaning that from the very start, there’s no question it’s going to go out to every facility and the same infrastructure holds at every facility. I think it’s different with health systems, where you’ve accumulated a patchwork of hospitals. They may each have their own IT staff, and legacy systems, and different service agreements, and Chiefs of Staff. For us, we’re very used to all hospitals operating under the same umbrella. So once you build a solution, for one, it’s just a matter of unlocking it for all the others.”

Technical Leader

Step 5. Design for and test in the expanded setting: 

• Ensure evidence of AI product safety and effectiveness before deciding to integrate an AI product in an expanded setting.
• Repeat guidelines under ‘Develop success measures’ and ‘Design AI solution workflow’ in the expanded setting to develop threshold measures and test AI product integration.
• Use the collected evidence to decide to clinically integrate the AI product in the expanded environment or to abandon the effort.

Step 6: Communicate with key stakeholders throughout the process

• Collect input and coordinate assessments, testing and expansion possibilities with relevant stakeholders in existing and expanded settings
• Standardizing systems of evaluation will allow institutions to more easily compare the use and efficacy of an AI product across settings.

Step 7: Update relevant AI product documentation, workflow, and training material

• Expanded use of an AI product should be reflected in an updated scope of use, workflow design, and training material.