A machine learning approach to web page filtering using content and structure analysis

Michael Chau, Hsinchun Chen

Research output: Contribution to journalArticle

105 Scopus citations

Abstract

As the Web continues to grow, it has become increasingly difficult to search for relevant information using traditional search engines. Topic-specific search engines provide an alternative way to support efficient information retrieval on the Web by providing more precise and customized searching in various domains. However, developers of topic-specific search engines need to address two issues: how to locate relevant documents (URLs) on the Web and how to filter out irrelevant documents from a set of documents collected from the Web. This paper reports our research in addressing the second issue. We propose a machine-learning-based approach that combines Web content analysis and Web structure analysis. We represent each Web page by a set of content-based and link-based features, which can be used as the input for various machine learning algorithms. The proposed approach was implemented using both a feedforward/backpropagation neural network and a support vector machine. Two experiments were designed and conducted to compare the proposed Web-feature approach with two existing Web page filtering methods - a keyword-based approach and a lexicon-based approach. The experimental results showed that the proposed approach in general performed better than the benchmark approaches, especially when the number of training documents was small. The proposed approaches can be applied in topic-specific search engine development and other Web applications such as Web content management.

Original languageEnglish (US)
Pages (from-to)482-494
Number of pages13
JournalDecision Support Systems
Volume44
Issue number2
DOIs
Publication statusPublished - Jan 2008

    Fingerprint

Keywords

  • Link analysis
  • Machine learning
  • Web mining
  • Web page classification

ASJC Scopus subject areas

  • Management Information Systems
  • Information Systems
  • Information Systems and Management

Cite this