Nonparametric Approaches to Comparing the Accuracy of Diagnostic Tests with Multiple Readers

In diagnostic imaging studies, the test results often depend on the subjective interpretation of the reader. Because of variability in readers' accuracy, studies evaluating diagnostic tests usually involve multiple readers. Receiver Operating Characteristic (ROC) analysis has been a popular method for evaluating the performance of diagnostic imaging modalities. In this mini-review, I introduce current literature on nonparametric methods to compare the accuracy of diagnostic tests with multiple readers using ROC analysis. Nonparametric approaches do not require distributional assumptions for the test results or the ROC curve, making them attractive for use when the total sample size/number of readers is small or when distributional assumptions may be problematic.