Data Deduplication – Overview and Implementation

Abstract It is noticeable globally that the rate at which information is stored and transferred has improved greatly. With computers and other multimedia in wide use and the explosion of the internet, the amount of data available exploded as well and there is also massive data transfer among individual and networks. Therefore there is need to think of how to save cost of buying storage devices and find a way of transferring information with less cost and reduce network bandwidth. Data deduplication is technique for eliminating duplicate copies of repeating data; it is more intelligent and specialized data compression. This technique is used to improve storage utilization and can also be applied to network data transfers to reduce the number of bytes that must be sent. This paper depicts various kind of data deduplication available. This paper depicts various kind of data deduplication available. The objective is to present enough detail about data deduplication to enable application and implementation of file-level deduplication. The system designed is a file-level (or single instance storage), post process deduplication to identify files that have similar content in the storage, state their location or path. The user can select the storage device to search. It is designed on windows 8 using java programming language and netbeans 8.0 IDE. With this system deployed it makes it possible to easily identify and remove redundant in repository. Further work is to form GUI for chunk or block-level deduplication in order to be able to select the threshold value for the block or chunk.