Removing Dedepulication Using Pattern Serach Suffix Arrays

With the increase of de-duplication in data sets of voter card or pan card, removing the de-duplication is the major challenge. Record linkage is the process of matching records from several databases that refer to the same entities. When appliedon a single database, this process is known as de-duplication. In this paper the investigation is done to how to remove the de-duplication with the help of suffix arrays. Suffix array is well organized data structure for pattern searching. This paper covers similarity metrics that are commonly used to spot similar field entries, and present a widespread set of duplicate detection algorithms that can identify almost duplicate records in a database. It also covers multiple techniques for improving the effectiveness and scalability of estimated duplicate detection algorithms. Finally, based on the algorithms, the paper presents how to remove the de-duplication from dataset.