Artificial intelligence (AI) has provided a promising solution for streamlining COVID-19 diagnoses and future public health crises. However, concerns surrounding security and trustworthiness impede the collection of large-scale representative medical data, posing a challenge for training a model that can be used worldwide.
An international research team, led by the University of Cambridge and the Huazhong University of Science and Technology, used a technique called federated learning to build an AI model they say can diagnose COVID-19 (or any disease that can be diagnosed using a CT scan) as well as a panel of professional radiologists. Using federated learning, an AI model in one hospital or country can be independently trained and verified using a dataset from another hospital or country, without data sharing.
According to the University of Cambridge, the researchers based their model on more than 9,000 CT scans from approximately 3,300 patients in 23 hospitals in the UK and China. Their results, reported in the journal Nature Machine Intelligence, provide a framework where AI techniques can be made more trustworthy and accurate, especially in areas such as medical diagnosis where privacy is vital.
In the early days of the COVID-19 pandemic, many AI researchers worked to develop models that could diagnose the disease. However, many of these models were built using low-quality data, ‘Frankenstein’ datasets, and a lack of input from clinicians. Many of the same researchers from the current study highlighted that these earlier models were not fit for clinical use in the spring of 2021.
“AI has a lot of limitations when it comes to COVID-19 diagnosis, and we need to carefully screen and curate the data so that we end up with a model that works and is trustworthy,” said co-first author Hanchen Wang from Cambridge’s Department of Engineering. “Where earlier models have relied on arbitrary open-sourced data, we worked with a large team of radiologists from the NHS and Wuhan Tongji Hospital Group to select the data, so that we were starting from a strong position.”
The researchers used two well-curated external validation datasets of appropriate size to test their model and ensure that it would work well on datasets from different hospitals or countries.
“Before COVID-19, people didn’t realize just how much data you needed to collect in order to build medical AI applications,” said co-author Michael Roberts, a senior research associate of applied mathematics at the University of Cambridge, as well as a postdoctoral fellow at AstraZeneca. “Different hospitals, different countries all have their own ways of doing things, so you need the datasets to be as large as possible in order to make something that will be useful to the widest range of clinicians.”
The researchers based their framework on 3D CT scans. They used 9,573 CT scans from 3,336 patients collected from 23 hospitals located in China and the UK. They also had to mitigate for bias caused by the different datasets, and used federated learning to train a better generalized AI model, while preserving the privacy of each data centre in a collaborative setting.
For a fair comparison, the researchers validated all the models on the same data, without overlapping with the training data. The team had a panel of radiologists make diagnostic predictions based on the same set of CT scans, and compared the accuracy of the AI models and human professionals.
The researchers say their model is useful not just for COVID-19, but for any other diseases that can be diagnosed using a CT scan. “The next time there’s a pandemic, and there’s every reason to believe that there will be, we’ll be in a much better position to leverage AI techniques quickly so that we can understand new diseases faster,” said Wang.
“We’ve shown that encrypting medical data is possible, so we can build and use these tools while preserving patient privacy across internal and external borders,” said Roberts. “By working with other countries, we can do so much more than we can alone.”
The researchers are now collaborating with the newly-established WHO Hub for Pandemic and Epidemic Intelligence, to explore the possibility of advancing the privacy-preserving digital healthcare frameworks.