Using genetic algorithm in building domain-specific collections: An experiment in the nanotechnology domain

Jialun Qin, Hsinchun Chen

Research output: Contribution to journalConference article

7 Scopus citations


As the key technique to build domain-specific search engines, focused crawling has drawn a lot of attention from researchers in the past decade. However, as Web structure analysis techniques advance, several problems in traditional focused crawler design were revealed and they could result in domain-specific collections with low quality. In this work, we studied the problems of focused crawling that are caused by using local search algorithms. We also proposed to use a global search algorithm, the Genetic Algorithm, in focused crawling to address the problems. We conducted evaluation experiments to examine the effectiveness of our approach. The results showed that our approach could build domain-specific collections with higher quality than traditional focused crawling techniques. Furthermore, we used the concept of Web communities to evaluate how comprehensively the focused crawlers could traverse the Web search space, which could be a good complement to the traditional focused crawler evaluation methods.

Original languageEnglish (US)
Number of pages1
JournalProceedings of the Annual Hawaii International Conference on System Sciences
StatePublished - Nov 10 2005
Externally publishedYes
Event38th Annual Hawaii International Conference on System Sciences - Big Island, HI, United States
Duration: Jan 3 2005Jan 6 2005


ASJC Scopus subject areas

  • Engineering(all)

Cite this