计算密集型海量数据查询处理关键技术分析

随着大数据时代的到来,计算密集型的海量数据查询的应用范围越来越广,在大数据环境下,从数据结构的特点出发实现分布式存储架构,并以高效查询为目的实现非结构化数据的索引机制,同时处理和优化聚集查询操作,分析与平台无关的并行编程模型。     

一种异构网络中的高效路由P2P覆盖网的设计

P2P覆盖网中存在着拓扑不匹配、负载不均衡等问题,会对资源查询效果造成不可忽视的影响。本文提出在节点度数为幂率分布的异构网络中实现高效路由的三层覆盖网设计思想,设计相关的节点分类、成簇和地址分配方法。针对查询路径长度、负载均衡程度、查询成功率等多个性能指标展开分析,并通过仿真实验验证了相比以往的层次化覆盖网,本文提出的方法在多个方面具有明显的优越性。     

一种面向智能交通应用的海量空间数据存储及并行处理方法

文中以智能交通应用中海量空间数据存储为研究对象,使用基于HDSF的分布式存储金字塔模型对空间数据进行分块;引入基于空间数据分类和删除技术的副本存储策略,将同一类影像存储在同一节点或者相近节点,降低连续访问延时;引入Spark作为空间信息查询的并行处理工具;利用RDD的快速存储机制,减少查询过程对I/O的读写,提高查询效率。实验结果证明了文中方法的有效性。     

面向系统级能耗优化的无线实时网络系统任务调度机制

传统的能耗优化方法通常利用CPU的DVFS协议以及射频通信休眠调度来减少耗能。却很少有工作在对系统任务负载的调度和运行模式分配的同时考虑计算/通信两个子系统对能耗的综合优化。此外,传统的能耗优化机制通常只考虑系统的总能耗最优,而忽略了系统中个别节点能源耗尽对网络连通性所产生的影响。提出一种以系统寿命最大化为目标的任务调度和工作模式分配机制。该机制根据任务的实时性约束条件和节点的剩余能量调度任务,通过控制DVFS协议给子任务设定工作频率,同时利用射频休眠调度来实现系统能耗和寿命的折中平衡。仿真实验的结果表明,任务调度与分配算法在无线嵌入式网络系统的能耗和寿命表现上优于传统的方法。     

物联网环境下的重点车辆监控海量信息处理研究

重点车辆监控是借助物联网技术,通过感知设备对重点监控车辆进行识别、定位、跟踪、监控和管理。重点车辆监控平台需面对海量数据的处理问题．云计算信息处理模式为重点车辆信息管理提供了新思路。借助云计算的概念,提出基于Hadoop分布式并行编程框架的重点车辆信息管理模式和基于MapReduce编程模型的重点车辆海量信息处理方案。然后,采用广东省的重点车辆监控平台的历史数据作为样本,搭建基于Hadoop的重点车辆海量信息处理测试平台。实验结果表明,该平台能够实现物联网环境下的重点车辆海量数据的快速处理与查询．满足实时监控的要求。     

基于子网连接点分布的分布式交通分配方法

提出多点连接双子网分布式分配的方法,将整网分为具有共同连接点的子网．基于一般交通分布,分配组合模型及其网络变换解法,提出连接点吸引测度的具体度量方法。OD点在不同子网的跨网需求的分配转化为寻找连接点分布,使在该分布下,各子网平衡可达整网平衡。由此,分布式分配转化为2个部分：连接点分布,子网分配组合计算以及基于连接点分布的子网分配。分析表明,在此分布式分配中,各子网既独立完成各自的分配任务又通过在迭代过程中相互传递阻抗信息和连接点分布信息来协同工作,具有分布式系统的优点,而且计算结果表明分布式分配是可行的。     

基于工作强度均衡的地铁乘务轮班计划优化模型

地铁乘务轮班计划是运营组织工作的重要组成部分,对于提高运营水平和降低运营成本有重要影响.本文采用基于“轮班单元”的周期循环编制模式,以工作强度均衡为目标构建乘务轮班划分和分配两阶段模型,划分模型中不区分同种类型的班次,分配模型将具体班次分配到划分结果中.采用粒子群算法与模拟进化算法相结合的混合智能算法求解,并针对乘务计划两阶段模型分别进行算法设计.最后以北京市某地铁线路数据为例进行求解,对比发现,本文模型求解方案可有效提高乘务员工作量均衡程度.     

基于改进萤火虫算法的云计算负载均衡研究

提出了改进的萤火虫优化算法,运用于解决云环境下资源负载均衡的问题.该算法改进了决策域半径的更新,能够有效地克服精确度不高、后期收敛较慢的缺点.利用这个优势,全面地考虑资源节点的负载指标,建立虚拟化资源管理负载模型,更改目标函数,实现云计算资源的平均负载.通过仿真验证,表明该算法能够提高资源利用率,可以在较短的时间内达到较好的负载均衡效果.     

基于簇形均衡评估的高速公路收费数据聚类

高速公路收费数据是一种高维、海量、分布特征未知的数据集,因此难以选择 何种算法和参数最适合此类数据的聚类.针对此问题,提出一种基于簇形均衡的聚类评估 指标IBCS,对各簇的形状、分布、密度和尺寸等多种形态进行均衡综合评估.该指标根据 数据集稀疏程度自适应调整邻域置信区间来度量簇结构的分散度和分离度;度量密度使 得IBCS 具有面向数据集的算法选择能力;度量簇大小避免簇划分过于悬殊的问题.UCI 数据集上多种候选算法评估比较实验验证了该指标灵活有效,能获得准确簇数并合理划 分.最后,基于IBCS 评估的西宝高速公路收费数据聚类结果表明,采用K-means 算法,簇 数为5时聚类模式最佳.     

基于快速一致性的微电网分布式控制策略

在微电网传统下垂控制中,针对系统负荷变化时所引起的频率波动及功率分配不合理的问题,提出了一种基于快速一致性的分布式控制策略.该策略以下垂控制为基础,增加了二次频率控制层和三次功率控制层.采用多智能体快速一致性算法,收敛得到平均功率,计算出所需的全局变量,补偿频率偏差,使各分布式电源按备用功率比例分担系统增加的负荷,并修正下垂系数,恢复系统频率、电压稳定以及进行功率的最优分配.最后,通过Matlab/Simulink仿真软件搭建了4节点微电网模型,验证并分析了该控制策略的有效性,使收敛时间缩短了近40%,提高了系统的响应时间.     

Numeric-based XML labeling schema by generalized dynamic method

Most efficient indeces and query techniques over XML (extensible markup language) data are based on a certain labeling scheme, which can quickly determine ancestor-descendant and parent-child relationship between two nodes. The current basic labeling schemes such as containment scheme and prefix scheme cannot avoid relabeling when XML documents are updated. After analyzing the essence of existing dynamic XML labels such as compact dynamic binary string (CDBS) and vector encoding, this paper gives a common unifying framework for the numeric-based generalized dynamic label, which can be implemented into a variety of dynamic labels according to the different user-defined value comparison methods. This paper also proposes a novel dynamic labeling scheme called radical sign label. Extensive experiments show that the radical sign label performs well for the initialization, insertion and query operations, and especially for skewed insertion where the storage cost of the radical sign label is better than that of former methods.     

Integrity Based Access Control Model for Multilevel XML Document

XML's increasing popularity highlights the security demand for XML documents. A mandatory access control model for XML document is presented on the basis of investigation of the function dependency of XML documents and discussion of the integrity properties of multilevel XML document. Then, the algorithms for decomposition/recovery multilevel XML document into/from single level document are given, and the manipulation rules for typical operations of XQuery and XUpdate QUERY, INSERT,UPDATE, and REMOVE, are elaborated. The multilevel XML document access model can meet the requirement of sensitive information processing application.     

UML Galaxy Diagram： An Approach of Conceptual Data Integration for OLAP

IntroductionWith the development of contemporary elec-tronic commerce,the multidimensional data analy-sis for On-Line Analytical Processing(OLAP)onthe web makes it necessary to integrate various da-ta sources such as Extensible Markup Language(XML)data and relational data on the conceptuallevel.XML is quickly becoming the new standardfor data representation and exchange on the WorldWide Web[1].At present,the most widespreadconceptual model is the Unified Modeling Lan-guage(UML)which …     

内核网络子系统的模块化设计

将面向对象的设计方法引入内核设计中,采用"低层处理"的策略,在多平面软件框架的数据平面内将所有的网络操作处理模块化;在内核空间中,研究并设计了虚拟网络块(VNB)框架.VNB框架由一系列的模块节点和节点之间的连接组成,数据流沿着模块节点之间的连接在不同的节点之间双向流动,当一个模块节点接受到数据包,执行处理后将数据包传递给另外一个节点模块.该框架基于UNIX的V流设计方法,采用"Netgraph"技术实现了复杂网络功能的处理高效率和设计灵活性.     

An authenticated group key distribution scheme for wireless sensor networks

In wireless sensor networks (WSNs), group key distribution is the core of secure communications since sensor nodes usually form groups and cooperate with each other in sensing data collection and in-network processing. In this paper, we present a scalable authenticated scheme for group key distribution based on a combinatorial exclusion basis system (EBS) for efficiency and one-way hash chains for authentication. The proposed scheme guarantees a lightweight authenticated group key updating procedure and is efficient in terms of storage, communication and computation overheads. Foundation item: the National High Technology Research and Development Program (863) of China (Nos. 2006AA01Z436, 2007AA01Z455, and 2007AA01Z473).     

Design and implementation of a data-driven dynamical reconfigurable cell array

The nature of dataflow computation demands the heavy flow of tokens amongst computation nodes. Traditional reduced instruction-set computer (RISC) processors are not suitable for such style computation. Devices that use long wire buses are not suitable for dataflow either. Reconfigurable computing devices (RCDs) consist of data transfer wires and computing resources. With minor modifications, reconfigurable cells can be adopted to perform dataflow computation. A reconfigurable cell array (RCA) is presented in this paper and it is suitable for dataflow computation. This cell array has a dynamic reconfigurable storage model. The distinctive features of the architecture include dataflow reconfigurable cells and reconfigurable storage. Dataflow applications can be mapped easily and effectively onto the cells. Reconfigurable storage is mainly used to manage data access and transmission. Furthermore, computation and data management are separated. Meanwhile, dynamical reconfiguration is accomplished, when some clusters of cells work in configuration mode and other clusters work in computation mode. The dataflow graphs of some algorithms are mapped onto our architecture, and the performance results are compared with those of CPU and GPU.     

基于四叉树孤立分割和屏幕误差的地形LOD算法

为解决静态和动态细节层次模型存在的数据冗余度大、精度判断标准单一和层次切换跳跃感强的问题,提出了基于四叉树孤立分割和屏幕误差的地形LOD(level of detail)算法.采用该算法,针对于规则格网,通过地形瓦片分割和数据预处理减少实时阶段计算量,利用四叉树孤立分割消除结点间依赖关系,并构建保守性屏幕误差评价标准以弱化视觉跳跃感,最后采用添加拆分点和高程平均值法消除相邻瓦片和结点间裂隙.实验结果表明:该算法能较好解决常规方法中存在的问题;可满足大规模地形实时三维显示的要求;实时显示计算量小,帧速可保持在0.03 s以内.      

Excel表格在土的液塑限试验数据处理中的应用

传统的液塑限联合测定试验中数据处理多采用作图法,人工计算工作量特别大,精度难以保证。运用Excel表格的强大数据处理功能,一次进行程序编制,可供计算多次测定数据,大大减轻了计算量,并且提高了精度,从而方便了数值计算。     

Solving airlines disruption by considering aircraft and crew recovery simultaneously

When disruptions occur, the airlines have to recover from the disrupted schedule. The recovery usually consists of aircraft recovery, crew recovery and passengers’ recovery. This paper focuses on the integrated recovery, which means above-mentioned two or more recoveries are considered as a whole. Taking the minimization of the total cost of assignment, cancellation and delay as an objective, we present a more practical model, in which the maintenance and the union regulations are considered. Then we present a so-called iterative tree growing with node combination method. By aggregating nodes, the possibility of routings is greatly simplified, and the computation time is greatly decreased. By adjusting the consolidating range, the computation time can be controlled in a reasonable time. Finally, we use data from a main Chinese airline to test the algorithm. The experimental results show that this method could be used in the integrated recovery problem.     

Scheduling for Uncertain Data Broadcast in Mobile Networks

With the increasing popularity of wireless sensor network and GPS (global positioning system) , uncertain data as a new type of data brings a new challenge for the traditional data processing methods. Data broadcast is an effective means for data dissemination in mobile networks. In this paper, the definition of the mean uncertainty ratio of data is presented and a broadcasting scheme is proposed for uncertain data dissemination. Simulation results show that the scheme can reduce the uncertainty of the broadcasted uncertain data effectively at the cost of a minor increase in data access time, in the case of no transmission error and presence of transmission errors. As a result, lower uncertainty of data benefits the qualities of the query results based on the data.