| BBS舆情系统爬虫模块的研究 |
| |
| 引用本文： | 张旭,张振江,刘云.BBS舆情系统爬虫模块的研究[J].铁路计算机应用,2010,19(12):18-21. |
| |
| 作者姓名： | 张旭 张振江 刘云 |
| |
| 作者单位： | 北京交通大学,通信与信息系统北京市重点实验室,北京,100044 |
| |
| 基金项目： | 国家自然科学基金资助项目，北京市自然科学基金资助项目，科技人员服务企业项目，教育部哲学人文社会科学研究重大课题 |
| |
| 摘 要： | 针对重要的舆论平台BBS论坛,提出利用网络爬虫构建适合抓取论坛数据的爬虫模块.通过分析BBS论坛以及比较不同类别爬虫的特点,介绍BBS爬虫模块的工作流程以及需要选择的策略,讨论URL的处理与重定向问题的解决方法,并提出对不同论坛模块定制不同任务的抓取策略.
|
| 关 键 词： | 舆情系统 聚焦爬虫 爬虫模块 定制任务 |
| 收稿时间： | 2010-12-15 |
Research on crawler module of BBS Public Opinion System |
| |
| ZHANG Xu,ZHANG Zhen-jiang,LIU Yun.Research on crawler module of BBS Public Opinion System[J].Railway Computer Application,2010,19(12):18-21. |
| |
| Authors: | ZHANG Xu ZHANG Zhen-jiang LIU Yun |
| |
| Institution: | (Key Laboratory of Communication ＆ Information Systems,Beijing Municipal Commission of Education Beijing Jiaotong University,Beijing 100044,China) |
| |
| Abstract: | In this paper,a web crawler module using web crawler was proposed to acquire large collections of web pages in the BBS.Based on the characteristics of BBS and the differences between crawlers,this work described the workflow and work strategy of the crawler module.The process and redirection of the URL was discussed.Furthermore,the idea of using different strategy to collect pages from different part of the BBS was also proposed. |
| |
| Keywords: | Public Opinion System focused web crawler crawler module custom tasks |
| 本文献已被 维普 万方数据 等数据库收录！ |
| 点击此处可从《铁路计算机应用》浏览原始摘要信息 |
| 点击此处可从《铁路计算机应用》下载免费的PDF全文 |
