Search Engine Big Data Management and Computing

Web Search Engine System includes wide range of in-memory data management of both data storage systems as well as Dynamic data processing techniques. Growing main memory capacity has fueled the development of in-memory big data management and processing. By eliminating disk I/O bottleneck, it is now possible to support interactive data analytics. Data gathered from crawling resources are spilt into Text and Numeric data and link need to be provided based on Offset Indexing. Frequent access data need to be loaded into Main memory using cache technologies such as web-cache in Apache or IIS. Bulk data are stored in In-memory databases such as REDIS. Big Numeric data can be stored in Flat File Binary Format such that they can be loaded into RAM in a single read operation. As run time disc access in flat file binary files are faster than conventional files, big data are stored in terms of Binary format which reduces the storage space up to 35% than other file formats. Some issues such as fault-tolerance and consistency are also more challenging to handle in in-memory environment. A Comprehensive presentation of important technology and factors to achieve efficient in-memory data management and processing in Web Search Engine are also considered.