DETERMINATION OF PACKED AND ENCRYPTED DATA IN EMBEDDED SOFTWARE

Subject of Research. Embedded software research for security faults can be handicapped by various anti-debugging techniques (encryption) and code wrappers (compression). The paper presents an overview of existing tools for definition of anti-debugging techniques. The disadvantages of existing solutions lie in the use of signature-based methods for analysis of executable files, that limits the scope of their application to the number of the known signatures. The existing statistical tests based on the entropy analysis of files give an ambiguous result. To determine the data conversion technique, a method is proposed for detection of packed and encrypted data in an executable firmware file. Method. The embedded software is represented as a finite sequence of bytes, where each byte can take one of 256 possible values. The proposed method combines the approaches based on the use of Pearson's chi-squared test to check the hypothesis of a uniform distribution of bytes in a file, as well as the use of the Monte Carlo method to approximate the number π in order to calculate the characteristics of the distribution of bytes in a file. The higher approximation accuracy of the number π and the closer the distribution of bytes in the file to a uniform one is, the more likely is the application of encryption algorithms for data transformation. Main Results. It is shown that the proposed criteria are more sensitive to deviations of a uniformly distributed random variable than the entropy analysis. Applying of these approaches to an experimental sample of files with various sizes, which were compressed/encrypted with a variety of algorithms, have shown correlations, that with a high degree of confidence give the possibility to state which algorithm (compression or encryption) the embedded software was subjected to. Practical Relevance. An approach is presented for determination of packed and encrypted data obtained as a result of the use of various anti-debugging techniques. The proposed method is applicable both in the analysis of malicious software and in the search and identification of security defects in embedded software.