An Evolving Approach on Efficient Web Crawler using Fuzzy Genetic Algorithm

Today, Web has become one of the largest and most readily accessible repositories and a rich resource of human knowledge. The traditional search engines index only surface Web whose pages are easily found. The focus has now been moved to invisible Web or hidden Web, which consists of a large warehouse of useful data such as images, sounds, presentations and many other types of media. To use such data, there is a need for specialized technique to locate those sites as we do with search engines. This paper focuses on an effective design of a Web Crawler that can optimally discover pages from the by employing fuzzy based genetic algorithm. The fitness function was evaluated using fuzzy fitness finder to produce best population for each iteration. A framework is used to discover the resource discovery problem and the results show the improvement in the crawling strategy and harvest rate.