Diagnosis, prognosis & prediction
-
IPD meta-analysis projects are also important for addressing questions about diagnosis, prognosis & prediction.
​
-
These allow a more detailed assessment about
​
-
Below we provide guidance, and for further details see Chapters 15 to 17 of our book and various references.
​​​
Test accuracy research using IPD meta-analysis
​
-
Test accuracy research aims to evaluate the performance of a particular medical test, usually for screening or diagnostic purposes.
-
Meta-analysis is needed to summarise the accuracy of one or more tests across multiple studies, in terms of sensitivity and specificity, and positive and negative predictive values.
​
-
The use of IPD, rather than aggregate data, has important potential to improve test accuracy meta-analyses.
-
For example, it allows the standardisation of participant inclusion criteria, reference standards, and thresholds to define positive and negative test results; it also enables the examination of how participant-level characteristics are associated with test accuracy.
​
-
Key steps to conduct an IPD meta-analysis of test accuracy include defining the research question, searching and classifying relevant studies, examining risk of bias, requesting and cleaning IPD, performing meta-analysis, and interpreting results.
​
-
The QUADAS-2 tool can be used to examine each study’s risk of bias and applicability for the IPD meta-analysis research question.
​
-
A one-stage IPD meta-analysis model can be used to summarise sensitivity and specificity (or predictive values), with a Bernoulli distribution to model test results within studies, and a bivariate normal distribution to model between-study heterogeneity of the true logit sensitivity and logit specificity, and their between-study correlation.
​
-
The approach can be extended to include participant-level and study-level covariates, to identify factors associated with changes in test accuracy. To avoid aggregation bias, participant-level covariates should be centered by their mean value and an additional term for the mean value should be included.
​
-
For tests which give results as ordered categories or continuous measurements, a separate meta-analysis can be performed at each threshold of interest.
-
More sophisticated analyses that model multiple thresholds simultaneously may sometimes be warranted, especially when some studies do not provide their IPD but report results for a subset of thresholds.
​
-
The clinical utility of a test can be summarised by fitting a trivariate meta-analysis of sensitivity, specificity and prevalence, and subsequently deriving a decision curve that summarises a test’s net benefit across a range of plausible risk thresholds that define clinical action.
​
-
The bivariate model can also be extended to compare the accuracy of multiple tests; however, caution is needed if including studies with only a subset of the tests of interest, as this introduces indirect (across-study) comparisons which may be subject to study-level confounding.
​​Prognostic factor research using IPD meta-analysis
​
-
A prognostic factor is any variable associated with the risk of future health outcomes.
​
-
Primary studies to identify prognostic factors are abundant, but often have conflicting findings and variable quality. This motivates systematic reviews and meta-analyses to identify, evaluate and summarise the evidence for whether particular factors are prognostic.
​
-
Meta-analysis based on published aggregate data are severely limited, especially by poor and selective reporting of primary studies and inappropriate statistical analyses, including use of (data-driven) cut-points to categorise continuous variables and a lack of adjustment for existing prognostic factors.
​
-
IPD meta-analyses can help overcome these problems. In particular, by standardising the inclusion/exclusion criteria; harmonising the set of adjustment factors; and analysing continuous factors on their original scale whilst allowing for potential non-linear trends.
​
-
Such IPD meta-analyses require a clear research question framed using a PICOTS system, and a transparent search undertaken for eligible studies and datasets. Initiating a collaborative network of researchers may help identify relevant IPD from studies currently unpublished or in grey literature.
​
-
Before obtaining IPD, researchers should extract relevant information from each study, to help decide on their inclusion and whether IPD should be sought. Items from the CHARMS-PF checklist may be used to guide this process.
​
-
The applicability and risk of bias of IPD from identified studies can be checked, using items from the QUIPS and PROBAST tools.
​
-
IPD meta-analyses should primarily aim to summarise the adjusted prognostic effect of a particular factor, to establish the factor’s added prognostic value after adjustment for established prognostic factors.
​
-
Two-stage and one-stage IPD meta-analysis approaches are possible, and usually yield similar results.
-
Advantages of the two-stage approach include being more computationally feasible and naturally avoiding aggregation bias, whilst having flexibility to include studies that differ in design (e.g. case-control, cohort, case-cohort, etc.), their available set of adjustment factors, and their provision of IPD.
​
-
Non-linear trends can be examined in either one-stage or two-stage IPD meta-analyses, for example using fractional polynomials or splines; in the two-stage approach, this requires a multivariate meta-analysis in the second stage to summarise the multiple parameters defining the non-linear shape.
​
-
Small-study effects (potential publication bias) can be examined on a funnel plot, which is most convenient after a two-stage IPD meta-analysis. Asymmetry may signal publication bias concerns, but may also be due to the factors causing between-study heterogeneity.
​
-
REMARK and PRISMA-IPD can be used to guide the reporting of IPD meta-analysis projects for prognostic factor studies.
​​Clinical prediction model research using IPD meta-analysis
​
-
IPD meta-analysis projects offers novel opportunities for the development and validation of clinical prediction models that aid the management of individuals in terms of diagnosis and prognosis.
​
-
Careful steps are required to identify, obtain and clean IPD from relevant studies or data sources.
-
Before obtaining IPD, a data extraction phase is helpful to obtain relevant information from each study, to help researchers decide on their inclusion and whether IPD should be sought. Items from the CHARMS checklist should be used.
​
-
The eligibility and risk of bias of IPD from identified studies can be checked, using items from the PROBAST tools, both before and after IPD retrieval.
​
-
For existing models, an IPD meta-analysis may allow their performance to be externally validated across different populations, subgroups and settings.
​
-
Performance can be evaluated using univariate and multivariate meta-analysis models that summarise either calibration and discrimination of model predictions, or clinical utility in terms of net benefit.
-
It is important to summarise not only average performance, but also heterogeneity in performance, and so random-effects meta-analysis models are required.
​
-
Often existing models show poor predictive performance when tested or applied in other populations or settings than those used for model development. In this situation, IPD meta-analysis may allow researchers update or tailor the existing model equation to improve performance in particular populations or settings.
​
-
Sometimes IPD meta-analysis may be needed to develop an entirely new model. Then, suitable methods for model development are required, such as penalised estimation to adjust for overfitting.
-
Generalisability of the model’s predictive performance can be examined using an approach called internal-external cross-validation.
​
-
IPD meta-analysis methods for prediction model research are also applicable to other datasets involving clustering, including those from electronic healthcare records.
​
-
The TRIPOD-CLUSTER statement should be adhered to when reporting results from IPD meta-analysis projects for clinical prediction model research.