SmartCrawler: For Site Locating and Balancing

The number of web pages in Internet are changed day by day. Therefore relevant information searching is difficult task. Most of this data is hidden behind query forms which is interface to unexplored databases containing high quality structured data. Existing search engine has drawback that they cannot access and index this hidden data of the Web, in fact it is very hard to derive the unseen info. For this, we propose a framework, namely SmartCrawler, just to gather all the web interfaces in depth. Initially that is site locating, in this step, center pages are searched using search engines which is used to avoid visiting a large number of pages. For achieving precise results for a focused crawl, SmartCrawler gives ranks to websites to prioritize highly relevant ones for a given topic. The second step, adaptive link-ranking gives fast in-site searching by searching most relevant links. In third step, i. e. Web Navigation, Space complexity problem of other crawler are managed. The required relevant links are stored in database. Parsing of web pages is done in web navigation step. To reduce time in visiting some highly related links in unseen web directories, we have designed a link tree data