Published Paper: ROC Curves and Nonrandom Data Paper

Author: Jonathan A. Cook

Publication: Pattern Recognition Letters, 2017, 85(1): 35-41

Abstract: This paper shows that when a classifier is evaluated with nonrandom test data, ROC curves differ from the ROC curves that would be obtained with a random sample. To address this bias, this paper introduces a procedure for plotting ROC curves that are inferred from nonrandom test data. I provide simulations to illustrate the procedure as well as the magnitude of bias that is found in empirical ROC curves constructed with nonrandom test data. The paper also includes a demonstration of the procedure on (non-simulated) data used to model wine preferences in the wine industry.

Download the Paper


The economic research fellows and staff economists generate high-quality working papers that inform the oversight activities of the PCAOB and are disseminated to stimulate discussion and critical comment to the benefit of the public. Working papers are preliminary materials that have not been approved by the Board and reflect only the views of the author(s).