Malware Detection and Tracer Approach for Operating System

Modern computer systems are built on a foundation of software components from a variety of vendors. While critical applications might undergo extensive testing and evaluation procedures, the heterogeneity of software sources hazardous the integrity of the execution environment for these trusted programs. For instance, if an attacker can be able to merge an application exploit with privilege increase vulnerability, the Operating System (OS) can become corrupted. Mandatory Access Control (MAC) in a commercial operating system to handle malware problem is a challenge but also a capable approach. The firmest barriers to apply MAC to defeat malware programs are the incompatible and unusable problems in existing MAC systems. The aim of our study is to address these issues design a novel Efficient Malware Detection and Tracer design (EMDT) using Hidden Markov model, which incorporates intrusion detection and tracing in an operating system. In this proposed approach conceptually consists of three actions: tracing, detecting and restricting deduced intruders. The novelty of the proposed study is that it leverages light-weight intrusion detection and tracing techniques to automate security label configuration that is widely acknowledged as a tough issue when applying a MAC system in practice. The other is that, rather than controlling information flow as a traditional MAC does, it traces intruders and restricts only their significant malware behaviours, where intruders characterize processes and executables that are potential agents of a remote attacker. Our prototyping and testings on Windows operating system show that Tracer can effectively defeat all malware samples tested via blocking malware behaviours while not causing a significant compatibility problem.