A new consensus statement, dubbed the CONSORT-AI extension, lays out the rules of the road for clinical trial reports on interventions involving artificial intelligence (AI). The statement, which was published in Nature Medicine and written by an international working group, includes 14 new items for researchers to routinely include in their manuscripts when reporting on AI interventions. The statement calls on researchers who report on trials that include AI to fully explain the algorithm version, input and output data, integration into trial settings, expertise of the users, and the protocol for acting upon the AI system’s recommendations. The idea is to promote transparent reporting of AI interventions and to build on the checklist outlined in the CONSORT 2010 statement, which provides minimum guidelines for reporting randomized trials. That statement, which was originally introduced in 1996 and has been widely endorsed by medical journals, has been updated over the years. The CONSORT-AI extension (Consolidated Standards of Reporting Trials – Artificial Intelligence) was developed alongside a companion statement for clinical trial protocols, the SPIRIT-AI (Standard Protocol Items: Recommendations for Interventional Trials – Artificial Intelligence). “It has been recognized that most recent AI studies are inadequately reported and existing reporting guidelines do not fully cover potential sources of bias specific to AI systems. The welcome emergence of [randomized controlled trials] seeking to evaluate newer interventions based on, or including, an AI component… has similarly been met with concerns about the design and reporting,” the working group wrote. Checklist itemsThe checklist items span most elements of clinical trial reporting, from the title and abstract to the disclosure of funding. The working group recommends that researchers indicate that the study intervention involves AI or machine learning in the title and/abstract and that they specify the type of model. They should also state the intended use of the AI intervention in the context of the clinical pathway, including its purpose and its intended users (whether healthcare professionals or patients). Researchers should also report AI-specific information related to study participants, including the inclusion and exclusion criteria at the level of participants and input data. Additionally, they should describe how the AI intervention was integrated into the trial setting, including any onsite or offsite requirements. The consensus statement also has six requirements related to intervention information, centering around the version of the algorithm, how the input data were acquired and selected, how poor quality and unavailable input data were assessed and handled, whether there was human-AI interaction in the handling of input data, the output of the intervention, and how the AI intervention’s outputs contributed to decision-making. Harms were also addressed in the CONSORT-AI checklist. The working group called on researchers to describe results of any analysis of performance errors and how errors were identified. The consensus document also calls for stating whether, and how, the AI intervention and its code can be accessed, and if there are any restrictions on its access or reuse. Welcome guidance The CONSORT-AI statement is welcome guidance for industry on how AI and engineering teams can navigate during the early lifecycle of product development, according to Rabia T. Khan, MBA, PhD, who is experienced in AI and data-driven drug discovery. “I believe these requirements are essential and provide the necessary frameworks for innovators from non-medical disciplines to solve healthcare challenges,” Khan said. “Providing AI-specific requirements can assist young companies to plan their go-to-market strategy clearly by having a set of requirements they need to fulfill to commercialize their products.” Khan said that in her experience, varying data quality and structure across different health systems can create challenges in deploying AI algorithms at scale. That makes the consensus statement’s focus on data standards and recommendations to explicitly address how data is handled within algorithms “pivotal,” she said. Moving forward, regulation for AI algorithms should also address diversity within datasets and potential biases of algorithms, she said. One approach to handle this is the provision of synthetic datasets from regulatory bodies, Khan suggested, which would allow companies to evaluate the algorithms prior to clinical implementation. Nature Medicine
Source: https://www.raps.org/news-and-articles/news-articles/2020/9/consort-ai-sets-standards-for-reporting-on-artific
