An Elegant Fusion of Concurrent Crawling and Page Rank Technique for Spidering Websites

The World Wide Web is expanding day by day. With the great growth of the Web, it has become a massive challenge for the all-purpose single process crawlers (A crawler is a program that downloads and stores Web pages, often for a Web search engine) to locate the resources that are precise and relevant in an appropriate amount of time, so more enhanced and convincing algorithms are in stipulate. Thus it becomes vital to improve the crawling procedure, in order to finish downloading pages in a sensible amount of time. Web crawler which employs multi-processing to allow multiple crawler processes to run concurrently. We have proposed a resourceful concurrent crawler that is fusion of page rank and concurrent multi-process crawler, offering a means to efficiently crawl the Web and presenting a scalable solution that allows crawl speeds to be tuned as needed.