基于搜索引擎的有害信息监控系统的设计与实现

在对搜索引擎核心技术进行研究的基础上,设计并实现一种采用主动扫描探测方法进行有害信息监控的系统.基于bot包设计网络蜘蛛模块,实现对html、asp、php和jsp等网页的自动抓取;采用反向最大匹配和二级哈希散列算法,实现中文分词;开发信息索引模块,实现对网页的批量和增量索引;开发有害信息检索模块,实现有害信息监控及预警功能.最后通过集成各模块,实现有害信息监控系统.

Research and implementation of Bad Information Detection System based on search engine

Based on the research of the kernel technology of search engine, Bad Information Detection System was proposed and implemented through initiative scanning .Html, asp, php, jsp and other style Web pages could be found automatically from lnternet by using the spider module based on bot package. Chinese word segmentation was implemented with reverse- going maximum matching method and two level hash arithmetic.Information index module for batch and incremental indexing and information searching module for bad information detection and alarming were implemented. At last, Bad Information System was established through module integration.