FoCUS: Learning to Crawl Web Forums?
Journal: International Journal of Computer Science and Mobile Computing - IJCSMC (Vol.3, No. 9)Publication Date: 2014-09-30
Authors : V.Shanmugapriya; S.Krishnaveni;
Page : 105-120
Keywords : EIT path; forum crawling; ITF regex; page classification; page type; URL pattern learning; URL type;
Abstract
In this paper, we present Forum Crawler Under Supervision (FoCUS), a supervised web-scale forum crawler. The goal of FoCUS is to crawl relevant forum content from the web with minimal overhead. Forum threads contain information content that is the target of forum crawlers. Although forums have different layouts or styles and are powered by different forum software packages, they always have similar implicit navigation paths connected by specific URL types to lead users from entry pages to thread pages. Based on this observation, we reduce the web forum crawling problem to a URL-type recognition problem. And we show how to learn accurate and effective regular expression patterns of implicit navigation paths from automatically created training sets using aggregated results from weak page type classifiers. Robust page type classifiers can be trained from as few as five annotated forums and applied to a large set of unseen forums. Our test results show that FoCUS achieved over 98 percent effectiveness and 97 percent coverage on a large set of test forums powered by over 150 different forum software packages. In addition, the results of applying FoCUS on more than 100 community Question and Answer sites and Blog sites demonstrated that the concept of implicit navigation path could apply to other social media sites.
Other Latest Articles
- Thermophysical characterization of earth blocks stabilized by cement, sawdust and lime
- Strengthening of concrete beams by CFRP: Experimental study and finite element analysis
- Effect of the heat curing on strength development of ultra-high performance fiber reinforced concrete (UHPFRC) containing dune sand and ground brick waste
- A “new” design requirement: the mitigation of the progressive collapse of buildings under exceptional loading
- Excess paste method to formulate a self-compacting concrete
Last modified: 2014-09-08 23:08:09