Title Rapid reviews: defining, evaluating methods, and reducing screening burden using artificial
intelligence
Title (croatian) Brzi pregledi literature: definiranje, ocjenjivanje metoda i olakšavanje
probira korištenjem umjetne inteligencije
Author Candyce Hamel https://orcid.org/0000-0002-5871-2137
Mentor Beverley Shea https://orcid.org/0000-0002-7686-2585 (mentor)
Committee member Ljubo Znaor (predsjednik povjerenstva)
Committee member Tina Poklepović Peričić (član povjerenstva)
Committee member Lorainne Tudor https://orcid.org/0000-0001-8414-7664 (član povjerenstva)
Granter University of Split School of Medicine Split
Defense date and country 2021-07-28, Croatia
Scientific / art field, discipline and subdiscipline BIOMEDICINE AND HEALTHCARE Public Health and Health Care Epidemiology
Universal decimal classification (UDC ) 614 - Public health and hygiene. Accident prevention
Abstract Introduction: Systematic reviews are considered the gold standard in collating available
evidence related to a specific question and are used to inform policy for health care public
health. They are considered to be essential in producing trustworthy guidelines. However, they
are time- and resource-intensive undertakings which may not meet the timeline of stakeholders
and policy-makers when urgent answers are required. The aims of rapid reviews are to produce
evidence reviews in a timely manner, while maintaining rigorous and robust methods.
However, to date, the only consensus around a definition of a rapid review is that a formal
definition does not exist. Additionally, there is no standardized set of methods for rapid reviews,
nor is there a comprehensive review which has compiled empirically evaluated rapid review
methods and evaluated the impact of these abbreviated methods. The aim of this doctoral
dissertation was to: (i) identify how rapid reviews have been defined in the literature and
perform a thematic analysis of these definitions to identify the key themes; (ii) identify and
create a repository of empirically evaluated methods abbreviations, and identify any gaps in the
research; and (iii) evaluate the reduction in the screening burden and perform of an artificial
intelligence and active machine-learning tool in an online systematic review software.
Methods: RR definitions: A systematic scoping review identifying rapid reviews published
between 2017 and January 2019 was performed. Definitions of rapid reviews were extracted
verbatim from these rapid reviews and a thematic analysis was performed to identify the key
themes which should be included when defining a rapid review. RR methods: A systematic
scoping review identifying formally evaluated rapid review methods abbreviations published
from 1997 onward was performed. In order to create a comprehensive repository of rapid
review documents, additional studies (e.g., around guidance on conducting rapid reviews,
discussing terminology) were identified. All publications were divided into one of four main
categories based on the purpose of the publication. Those that formally evaluated rapid review
methods abbreviations were mapped to the Methodological Expectations of Cochrane
Intervention Reviews (MECIR) to determine if they met these criteria. Lastly, an experimental
evaluation was conducted in DistillerSR ® on 10 completed systematic reviews, using the
artificial intelligence simulation tool, to measure the reduction in the screening burden and
accuracy (i.e., how many relevant records were missed) when prioritized screening using active
machine-learning was employed.
26
Results: RR definitions: A total of 204 definitions that could be thematically analyzed were
identified in 216 rapid reviews and 90 rapid review methods papers. Eight major themes were
identified, with four themes found in 48.5% or more of the definitions: Theme 4: Compare and
contrast to SRs (68.1%; 139/204), Theme 2: Variation in shortcut methods (54.9%; 112/204),
with Theme 1: Accelerated/rapid process and Theme 6: Resource efficiency rationale tied
(48.5%; 99/204 each). This lead to a suggested definition of “A rapid review is a form of
knowledge synthesis that accelerates the process of conducting a traditional systematic review
through streamlining or omitting a variety of methods to produce evidence in a resourceefficient manner.” RR methods: Ninety rapid review methods papers were identified, of which
14 formally evaluated rapid review methods abbreviations addressing several, but not all, key
dimensions related to the conduct of a review. Only a cursory mapping to MECIR criteria was
possible, as insufficient information impeded the ability to determine if criteria were met.
Active machine-learning prioritization tool: The active machine-learning tool, employing
prioritized screening, greatly reduced the screening burden of the 10 systematic reviews that
were evaluated. The median percentage of studies required to be screened to identify 95% of
the records included at the title and abstract level (true recall @ 95%) was 47.1% (IQR: 37.5 to
58.0%). Among the 5% that were not yet identified as included (i.e., title and abstract false
negatives), none were included in the final review, resulting in 100% accuracy.
Conclusion: The emergence of rapid reviews, highlighted by the ongoing COVID-19
pandemic, requires consistency in how they are defined, in order to identify and produce a
homogenous set of products regardless of the term used to identify them. Producers of rapid
reviews also need guidance on which abbreviated methods may be used to keep potential bias
minimized. Lastly, active machine-learning is a viable method to reduce the screening burden
and was shown to be very accurate.
Abstract (croatian) Uvod: Sustavni pregledi smatraju se zlatnim standardom u prikupljanju dostupnih dokaza koji
se odnose na određeno pitanje i koriste se za informiranje politika javnog zdravstva. Smatraju
se ključnim u stvaranju pouzdanih smjernica. Međutim, izrada sustavnih pregleda zahtijeva
vrijeme i resurse, i možda neće biti napravljeni dovoljno brzo za dionike i donositelje odluka
kada su potrebni hitni odgovori. Ciljevi brzih pregleda (engl. rapid reviews, RR) su pravodobno
izraditi preglede dokaza, uz zadržavanje rigoroznih i robusnih metoda. Međutim, do danas je
jedini konsenzus oko definicije RR taj da formalna definicija ne postoji. Uz to, ne postoji
standardizirani skup metoda za RR, niti postoji sveobuhvatan pregled literature koji je
empirijski procijenio metode RR i procijenio učinak tih skraćenih metoda. Cilj ove doktorske
disertacije bio je: (i) utvrditi kako su RR definirani u literaturi i provesti tematsku analizu tih
definicija kako bi se prepoznale ključne teme; (ii) pronaći i napraviti repozitorij empirijski
procijenjenih skraćenih metoda za izradu RR te prepoznati područja u kojima su potrebna nova
istraživanja; i (iii) procijeniti može li se olakšati probir literature korištenjem umjetne
inteligencije i aktivnog alata za strojno učenje u internetskom računalnom programu za izradu
sustavnog pregleda.
Metodologija objedinjenih radova: Definicije brzih pregleda literature: Napravljen je
pretražni sustavni pregled (engl. scoping systematic review) kojim su nađeni RR objavljeni
između 2017. i siječnja 2019. godine. Definicije RR izvučene su doslovno iz tih RR i provedena
je tematska analiza kako bi se utvrdile ključne teme koje bi trebale biti uključene prilikom
definiranja RR.
Metode RR: Napravljen je pretražni sustavni pregled kojim su pronađene metode za skraćenje
RR, u radovima objavljenim od 1997. nadalje. Kako bi se napravio opsežan repozitorij
dokumenata o RR, pronađene su dodatne studije (npr. o smjernicama za provođenje brzih
pregleda, rasprava o terminologiji). Sve su publikacije podijeljene u jednu od četiri glavne
kategorije na temelju svrhe publikacije. Oni koji su formalno ocjenjivali skraćene metode za
RR mapirane su pomoću smjernica za pisanje Cochraneovih sustavnih pregleda MECIR (engl.
Methodological Expectations of Cochrane Intervention Reviews) kako bi se utvrdilo
ispunjavaju li te kriterije.
Na koncu je napravljena eksperimentalna evaluacija u programu DistillerSR® na 10 završenih
sustavnih pregleda korištenjem alata za simulaciju umjetne inteligencije kako bi se izmjerilo
24
smanjenje opterećenja i točnosti probira (tj. koliko je relevantnih zapisa propušteno) kada se
koristi probir pomoću strojnog učenja.
Rezultati: Definicije RR: U 216 RR i 90 članaka o metodama RR pronađene su ukupno 204
definicije koje se mogu tematski analizirati. Definirano je osam glavnih tema, a četiri teme
pronađene su u 48,5% ili više definicija: Tema 4: Usporedba i kontrast sa sustavnim pregledima
(SR) (68,1%; 139/204), Tema 2: Varijacije u skraćenim metodama (54,9%; 112/204), Tema 1:
Ubrzani / brzi postupak i Tema 6: Obrazloženje obrazloženja učinkovitosti resursa (48,5%;
99/204 svaka). To je dovelo do predložene definicije "RR je oblik sinteze znanja koji ubrzava
postupak provođenja tradicionalnog sustavnog pregleda putem racionalizacije ili izostavljanja
različitih metoda kako bi se brže došlo do potrebnih dokaza."
Metode RR: Pronađeno je devedeset radova o metodama RR, od kojih je 14 formalno ocijenilo
skraćene metode RR koje se odnose na nekoliko, ali ne sve, ključnih dimenzija povezanih s
provođenjem pregleda literature. Bilo je moguće samo površno mapiranje kriterija MECIR-a,
budući da su nedovoljne informacije priječile mogućnost utvrđivanja jesu li kriteriji
zadovoljeni.
Alat za prioritizaciju aktivnog strojnog učenja: Alat za aktivno strojno učenje, koji koristi
prioritetni probir, uvelike je smanjio teret probira za 10 sustavnih pregleda koji su procijenjeni.
Medijan postotka zapisa koje je trebalo pregledati kako bi se pronašlo 95% zapisa uključenih
na razini naslova i sažetka iznosio je 47,1% (interkvartalni raspon: 37,5 do 58,0%). Među 5%
koji nisu prepoznati kao uključivi (tj. lažno-negativni naslovi i sažeci), niti jedan nije bio
uključen u konačni pregled, što je dovelo do 100%-tne točnosti.
Zaključak: Pojava RR zahtijeva dosljednost u njihovom definiranju kako bi se u literaturi
mogao pronaći i napraviti homogeni skup proizvoda, bez obzira na termin koji se koristi za
njihovu identifikaciju. Autorima koji rade RR trebaju smjernice o tome koje se skraćene metode
mogu koristiti kako bi se potencijalna pristranost svela na najmanju moguću mjeru. Na koncu,
aktivno strojno učenje održiva je metoda za smanjenje opterećenja probirom literature, koja se
pokazala vrlo preciznom.
Keywords
artificial inteligence
systematic literature reviews
MEDLINE
public health policy
Keywords (croatian)
umjetna inteligencija
sustavni pregledi literature
MEDLINE
politika javnog zdravstva
Language english
URN:NBN urn:nbn:hr:171:149967
Promotion 2022
Study programme Title: Obtaining a doctorate of science outside of doctoral studies Study programme type: university Study level: postgraduate Academic / professional title: doktor/doktorica znanosti (doktor/doktorica znanosti)
Type of resource Text
File origin Born digital
Access conditions Open access Embargo expiration date: 2022-07-29
Terms of use
Created on 2022-01-25 13:09:25