期刊名称：International Journal of Grid and Distributed Computing
印刷版ISSN：2005-4262
出版年度：2015
卷号：8
期号：2
页码：49-60
DOI：10.14257/ijgdc.2015.8.2.06
出版社：SERSC
摘要：Currently, the efficiency of the existing focused crawlers is not high because of their unsatisfactory precision. In this article, we analyze the URL analysis methods of the existing focused crawlers, and propose a URL analysis algorithm based on the semantic content and link clustering in cloud environment. In this algorithm, the download URLs are clustered with the philosophy of clustering on the basis of VSM to improve the precision of the focused crawler according to the correlation between download URLs and new URLs. The algorithm is evaluated on Heritrix3.10 compared with Best First Search algorithm and Shark Search algorithm. The experiment results demonstrate that the algorithm proposed can collect web pages related to the given topic accurately and effectively.Moreover, the algorithm has a good ability of learning which proves the possibility of this algorithm.
