UNCERTAINTY ESTIMATION AND USAGE FOR DEEP LEARNING MODELS

The default DL approaches in ML tend to output only prediction, but not an uncertainty measure alongside the prediction. There are several approaches to DL model modification that allow deciding if the model can be trusted. The approaches vary by computational load and performance considering given constraints. In the real world project, it is often not possible to modify the model or perform retraining to apply common uncertainty estimation techniques (black box problem). In the first part of the paper, we aim to measure the uncertainty of the model in practice. We have researched tolerable perturbations as a way to enforce noise in the input data. A framework was built that acts as a compound for a prediction model in image classification tasks and allows output uncertainty for given samples. For test purposes, a CNN model will be used over the CIFAR-10 dataset to showcase uncertainty evaluation. We also show how to use uncertainty values to get data insights into a real-world task In the second part, we discuss how to get a model to know when prediction is uncertain. We built a selective classifier to increase the performance of the model by narrowing the confidence interval on the input data and used the aforementioned uncertainty estimations in the rejection classifier. To showcase classifier features, we made an experiment with a softmax-based uncertainty classifier (vanilla) and Dirichlet distribution based value. To measure the performance of the predictor, we took the Brain Tumor Classification (MRI)[6] dataset as an example. For the received predictors we measured coverage and selective risks. We have shown that one could get significant accuracy gains by using selective models given accurate uncertainty measure.