基于多种策略的页面内容提取算法

针对W eb页面存在与主题无关的噪音的问题,提出了基于页面结构与页面内容相结合的多策略页面内容提取算法.该算法根据改进的VIPS(基于视觉信息的页面分割算法)生成页面的块结构树,通过定义内聚度阈值和块结构树的最大深度,实现了块结构树中不同区域内不同分块粒度的要求;根据W eb页面提供的结构信息和内容信息提取块结构树叶子节点中的"主题"块和"主题相关"块;最后,对主题块和主题相关块的内容进行合并,提取页面的主要内容.实验表明,对任意下载、不同内容类型的页面,该算法都能有效地提取页面内容.

Web Content Extraction Based on Multiple Strategies

In order to filter the noise in a web page,a new multi-strategy algorithm to extract the contents of a web page was proposed.With this algorithm,the granularity in different areas of the block tree of a web page established by the improved VIPS(visual based page segment) algorithm is controlled by defining the permitted degree of coherence and the maximum depth of the block tree.In addition,"topic" or "topic-relevant" blocks among the leaves of the block tree can be extracted from the blocks' content information and structure information.Finally,the main content of a web page can be extracted by merging these blocks' contents.Experiments on the web pages of three sites indicates that the proposed algorithm is effective for extracting the contents of any type of web pages.