| 数据整合中交互过滤扫描算法 |
| |
| 引用本文： | 轩志远,陆介平,方伍元.数据整合中交互过滤扫描算法[J].江苏科技大学学报(社会科学版),2006,20(3):55-59. |
| |
| 作者姓名： | 轩志远 陆介平 方伍元 |
| |
| 作者单位： | 江苏科技大学,电子信息学院,江苏,镇江,212003;江苏科技大学,电子信息学院,江苏,镇江,212003;江苏科技大学,电子信息学院,江苏,镇江,212003 |
| |
| 摘 要： | 当重复记录检测算法对大型数据库进行处理时,由于需要多次扫描数据库,时空消耗较大且效率较低。为了提高检测算法的效率,在区域无关扫描算法的基础上提出了交互过滤扫描算法。该算法在关键字排序上提出了按键盘字母顺序的排序规则,可缩小了重复记录之间的距离,减少比较次数。在两条记录应用字符串模糊匹配算法进行比较前,根据用户提供的相似度对数据库中的记录进行过滤处理。可避免数据库中差异度较大的两条记录在模糊匹配中的时间消耗。实验证明该算法的性能要优于区域无关扫描算法。
|
| 关 键 词： | 数据挖掘 数据清理 重复项清理 过滤扫描 |
| 文章编号： | 1673-4807（2006）03-0055-05 |
| 收稿时间： | 2005-08-31 |
| 修稿时间： | 2005年8月31日 |
An Interaction Filter Scan Algorithm in Data Integrating |
| |
| XUAN Zhiyuan,LU Jieping,FANG Wuyuan.An Interaction Filter Scan Algorithm in Data Integrating[J].Journal of Jiangsu University of Science and Technology:Natural Science Edition,2006,20(3):55-59. |
| |
| Authors: | XUAN Zhiyuan LU Jieping FANG Wuyuan |
| |
| Institution: | School of Electronics and Information, Jiangsu University of Science and Technology, Zhenjiang Jiangsu 212003, China |
| |
| Abstract: | To promote the efficiency in scanning a large database,an ITFS(Interactive Filer Scan algorithm) for eliminating duplicate is proposed based on the domain-independent algorithm.The times of comparing two records can be decreased by using a new rule,which sorts the records in alphabet order on the keyboard.The distance between the duplicated records can be reduced.Furthermore,the ITFS filters the records in the database before the comparison with the complex record matching method in the scanning according to the threshold provided by the user in interactive manner.The time consumed in the record matching can be reduced and the efficiency will be improved.Results from experiments indicate that the performance of ITFS may precede the domain-independent algorithm. |
| |
| Keywords: | data mining data cleaning eliminating duplicate filter scan |
| 本文献已被 CNKI 维普 万方数据 等数据库收录！ |
|
