Deep Web Mining Using C# Wrappers

World Wide Web (Internet) has immense collection of information that can be extracted for building knowledge base and business intelligence purposes. Generally that valuable information lies deep inside web databases and is not accessible directly through surface web crawling methods. This information can only be accessed via a focused crawler or wrapper program customized for a particular website. The wrapper can submit a set of values for form fields and imitate user actions such as mouse click or link navigations as performed on a web browser, thus saving the response page received from a web server and can then after extract information such as table data, links, image URLs etc after parsing the DOM structure of the document. We propose a C# crawler that can crawl a basic website and a set of related procedures (wrapper) which can extract (or mine) data from that resource by making use of regular expressions (Regex) patterns.