ROC Curves and Nonrandom Data

Paper Author: Jonathan A. Cook
Publication: Pattern Recognition Letters, 2017, 85(1): 35-41

This paper is concerned with evaluating a model’s predictions when only nonrandom data are available. For example, the Internal Revenue Service (IRS) uses a model that predicts tax-filing errors to select tax returns for audits. The IRS may want to measure the predictive power of their statistical models with data on recent audits.

While there is a large literature in econometrics that studies problems related to nonrandom data, there appears to be no prior work that studies evaluating predictions with nonrandom data.

This paper shows that Receiver Operating Characteristic (ROC) curves, which are the most common tool for evaluating these types of predictions, provide a misleading picture of a model's predictive power when used with nonrandom data.

This paper also provides a procedure, which leans heavily on the econometric literature, that provides a consistent estimate of the ROC curve that would be obtained with random data. This procedure is illustrated with simulated data and an example with wine-quality data.

This paper’s results also apply to problems that arise in banking and recommender systems. When banks evaluate their probability of default models, they do not have a random sample of loans available. A recommender system (as used by popular online retailers and online dating websites) will only show the user items that are predicted to be of interest. In both of these examples, this paper’s procedure can provide a clearer picture of the model’s predictive power.