OBJECTIVES: Machine learning (ML) and artificial intelligence (AI) may enable automation of components of systematic reviews. These methods need evaluating to ensure continued robustness of systematic reviews and appropriate integration into existing workflows. The objective of this systematic review was to identify recent evidence of the use, performance, and implementation of ML/AI to support systematic reviews.
METHODS: We searched Embase, Medline, and the Cochrane Library to identify articles published since 1 January 2022. We included any study type that evaluated the performance of ML/AI tools and explored the real-world implementation of these tools. We synthesised results using a staged approach, and this abstract summarises findings from published literature reviews.
RESULTS: We included 20 articles corresponding to 19 literature reviews. These reviews described the use of ML/AI tools (n = 5), implementation evaluation of ML/AI (n = 3), performance evaluation of ML/AI tools (n = 5), identification of available ML/AI tools (n = 3), and ML/AI-assisted methods (n = 3). Most reviews, particularly those assessing performance, considered screening (n = 14) or data extraction (n = 14); limited evidence was found on filtering, searching, deduplication, snowballing, quality appraisal, and qualitative synthesis. Common performance metrics include recall, precision, and F1 scores; prioritisation of recall optimisation was recommended. Typically, ML/AI tools are trained on titles and abstracts obtained mainly from Medline and tend to focus on randomised controlled trials. Barriers to the uptake of ML/AI include lack of regulatory agency guidance, costs, training requirements, user friendliness of tools, and transparency concerns.
CONCLUSIONS: Further research is needed on the best approaches to maximise recall of ML/AI tools. There is a need to expand the data sources used for training ML/AI tools. Given high expectations for ML/AI, performance assessment against current practice, rather than perfection, may be appropriate to enable efficiency and increased uptake of these methods.
