Guides

Disseminate information to end users

Why does this matter?

Successful use of the AI product depends on the work environment and social interactions surrounding the technology, not only the performance of the technology itself. Proactive communication with end-users builds trust and buy-in. Communication channels need to be two-way and end-users need to be able to share feedback to rapidly identify challenges.

While it’s important that healthcare delivery settings are transparent with patients within the target population of AI products, most organizations do little on that front today.

Please note that while disseminating information about an AI product with end-users and affected patients promotes transparency, it does not ensure interpretability or explainability. To learn more about how and when to prioritize interpretability, please see the decision tree below:

Decision Tree Example

Scenario 1

Scenario: An AI product identifies a patient at high risk of condition X. A clinician confirms the risk assessment and deems that the patient is in need of intervention.
Question:
How much does the intervention depend on the values of individual data elements fed into the AI product?

Scenario 1.1

Scenario: The clinician does not need to understand any more about a patient to determine the appropriate intervention.

Priorities in modeling: Interpretability and explainability do not need to be prioritized.

Focus primarily on AI product accuracy.

Scenario 1.2

Scenario: The clinician needs to understand more about a patient to determine the appropriate intervention.
Question
Is there strong evidence supporting a narrow set of potential interventions?

Scenario 1.3

Scenario: The clinician needs to understand more about a patient to determine the appropriate intervention.
Question
Is there strong evidence supporting a narrow set of potential interventions?

Scenario 1.2.1

Question:
Is there evidence to support tailoring interventions to specific sub-groups meeting certain criteria?

Scenario 1.2.1.1

Priorities in modeling: Interpretability and explainability do not need to be prioritized.

Focus primarily on AI product accuracy.

Consult literature and clinical domain experts to build out expert heuristics used for tags. Present tags to end-users to support targeted interventions.

Scenario 1.2.1.2

Scenario: The clinician needs to understand more about a patient to determine the appropriate intervention. However, there is insufficient evidence to support tailoring of the intervention to specific sub-groups.
Priorities in modeling: Interpretability and explainability do not need to be prioritized.

Focus primarily on AI product accuracy.

Consult clinical domain experts to identify additional information to present users along with AI model output.

Expect clinicians to make tailored treatment recommendations independent of AI model output.

Scenario 1.2.2

Scenario: The clinician needs to understand more about a patient to determine the appropriate intervention. However, there is insufficient evidence to support a narrow set of interventions.
Warning: This may not be a suitable task for AI to address. This may represent an opportunity for enrolling patients identified by the AI product in a clinical trial to test different interventions.

Scenario 1.3.1

Question:
Is there evidence to support tailoring interventions to specific sub-groups meeting certain criteria?

Scenario 1.3.2

Scenario:The clinician needs to understand more about a patient to determine the appropriate intervention. However, there is insufficient evidence to support a narrow set of interventions.
Warning: This may not be a suitable task for AI to address. This may represent an opportunity for enrolling patients identified by the AI product in a clinical trial to test different interventions.

Scenario 1.3.1.1

Priorities in modeling: Interpretability and explainability do not need to be prioritized.

Focus primarily on AI product accuracy.

Consult literature and clinical domain experts to build out expert heuristics used for tags. Present tags to end-users to support targeted interventions.

Scenario 1.3.1.2

Scenario: The clinician needs to understand more about a patient to determine the appropriate intervention. However, there is insufficient evidence to support tailoring of the intervention to specific sub-groups.
Warning: This may not be a suitable task for AI to address. This may represent an opportunity for enrolling patients identified by the AI product in a clinical trial to test different interventions.

Submission

View Full Decision Tree

How to do this?

Step 1: Build relationships and establish communication channels across organizational stakeholders

  • Designate senior clinical and administrative leaders within the context of use to lead communication and marketing efforts. 
  • Designate a clinical champion within the context of use to serve as a single point of contact who is responsible for responding to feedback from front-line clinicians and affected patients during the AI product rollout
  • Develop a mechanism for front-line workers and patients affected by the AI product rollout to share feedback with the single point of contact. Communication mechanisms include:
    • Email addresses 
    • Online forms
  • Listen to the perspectives of front-line workers or patients who are hesitant to engage with the AI product. 

Step 2: Ensure access to information about the AI product 

  • Designate a technical lead who is familiar with the AI product and able to respond to questions during the AI product rollout.
  • If working with an outside vendor, require disclosures about relevant information to share with end-users and affected patients when doing a preliminary assessment of the procurement of the AI product (see the guide on assessing the quality of external AI product options).

Step 3: Develop content and dissemination channels targeted to affected clinicians 

  • Disseminate the following information to affected clinicians:
Name of DocumentFormatContent included
Problem description*1-page snapshot
  • Problem addressed by the AI product
  • Scope of use of the AI product
  • Performance report of the AI product
  • Workflow and user experience for different types of clinicians
  • Opportunities for improvement through effective use of the AI product
Model Facts Label**1-page factsheet
  • Instructions for model use within the local context
    Scope and indication for use
  • Warnings
  • Contact information for support
End-user manual2-page manual
  • Key functionality of the AI product
    Different scenarios of appropriate use
  • Any known failure modes
  • Instructions in the event that the AI product stops working or goes offline (i.e. alternate workflows)
* An example of a 1-page high-level snapshot can be found in Figure 3 of this manuscript. The target audience for this document is affected clinicians. The purpose of this document is to both motivate affected clinicians to effectively use the AI product and build trust among affected clinicians. 
**An example of a Model Facts label can be found here.
  • Continue to host updated information on a centralized web page accessible to all front-line workers affected by the AI product.

“We had posters up on the units, we had little cards, we would hand out, we have a YouTube video, we would just go in for services, you know, at shift changes, morning shifts or nights… and then identifying superusers who would be the ones that that would train other staff”

Technical Leader

Step 3: Launch a governance committee to meet every month during the pilot period

  • Define an initial pilot period for the roll-out. We recommend a minimum of a 3-month pilot period.
  • Establish a governance committee with a minimum of monthly meetings during the pilot period. This committee will convene stakeholders across disciplines and make decisions about necessary changes.
  • Designate a clinical lead to chair the governance committee.
  • Establish a charter for the governance committee that includes making adjustments to the AI product during the pilot period, updating and disseminating information about the AI product, and informing the decision at the end of the pilot period to either continue use, update, or decommission (see the guide on determine if updating or decommissioning is necessary)
  • Seek participation from the following stakeholders groups within the governance committee: a technical lead who is familiar with the AI product (if built internally) or with the modeling approach (if procured from an external vendor); administrative leaders who are responsible for the quality of care within the context of use;  and representatives from the end-user community and affected clinicians, including physician and nursing representatives. 

Step 4: Share updates and feedback about short-term outcomes with affected clinicians

  • Update affected clinicians on the following questions:
    • Is the new AI product and associated workflow working? 
    • If it is working in general, how is it working for patients? 
    • Are there opportunities to improve the care provided to patients? 
  • Provide general and clinician-specific feedback through qualitative and quantitative reports.

Step 5: Establish a process for sharing future updates to AI products and the work environment

  • Establish a process for sharing updates about the AI product with affected clinicians. Defining the process now will ease later steps in the lifecycle when changes or adaptations need to be made to the AI product or work environment.

adopt health ai