Performance analysis of full-reference objective image and video quality assessment metrics

This paper presents the performance evaluation of image and video quality assessment metrics on two publicly available datasets with subjective quality ratings. In addition to the performance analysis at the global level - at the level of complete datasets, the paper presents the objective measures performance evaluation on subsets of signals inside them. The image dataset contains five subsets created by using different types of JPEG compression, while the video dataset contains six subsets of sequences - four created by compression of original sequences, and two subsets are with video signal transmission characteristic degradations. To determine the success of objective measures, i.e. comparison of subjective and objective quality scores, there were used measures accepted by the International Telecommunication Union - ITU (linear correlation coefficient, rank-order correlation coefficient, mean absolute error, root mean squared error and outlier ratio). It was shown that objective quality measures can reach a high level of agreement with the results of subjective tests on subsets of datasets. Objective measures performances depend on the type of degradation which significantly affects the performance at the complete dataset level. The difference in performances is more pronounced on video sequences due to considerable visual differences in sequences created by using compression, packet losses and additive Gaussian noise. Therefore, we can say that a universal objective measure, i.e. measure that is useful for different types of signal degradation, for different degradation levels, and for different applications currently does not exist.