支持大规模监测数据分析的LaUD-KV聚集操作扩展

doi:10.13196/j.cims.2013.12.xuhao.3035.8.20131214

摘要/Abstract

摘要： 结合当前维护、维修和大修行业对于大数据存储和查询的需求,对当前工业数据存储方案和大数据查询模式的优缺点进行了研究和分析,提出一套在分布式存储环境下工业大数据的聚集函数查询框架和对应算法。基于现有的LaUD-KV存储模式和LaSQL语言,实现了一套能满足工业大数据查询需求的类SQL聚集函数,使得在数据过亿的情况下,系统对于聚集查询的响应时间缩短为原有的30%左右,同时为上层的维护、维修和大修监测平台提供了技术支持。

关键词: 维护、维修和大修, 监测数据, LaUD-KV, 聚集函数, 查询框架

Abstract: For the requirement of Maintenance,Repair and Overhaul/Operation (MRO) industry for storage and query of mass data,the advantages and disadvantages of existed storage scheme and query mode were studied.A new framework for aggregation functions and its corresponding algorithms base on distributed storage environment was proposed.On the basis of existing LaUD-KV storage engine and LaSQL query language,a set of SQL aggregation function which satisfied the industrial data query was realized.With more than billion data,the response time of system for aggregation query was reduced by 70%,and the technical support for upper MRO monitoring system was provided.

Key words: maintain, repair and overhaul/operation, monitoring data, LaUD-KV, aggregation function, query framework

中图分类号:

TP319

徐昊,杨义繁,张力,刘英博,王建民. 支持大规模监测数据分析的LaUD-KV聚集操作扩展[J]. , DOI: 10.13196/j.cims.2013.12.xuhao.3035.8.20131214.

XU Hao,YANG Yi-fan,ZHANG Li,LIU Ying-bo,WANG Jian-min. LaUD-KV aggregation function expansion for massive equipment monitoring data analysis[J]. , DOI: 10.13196/j.cims.2013.12.xuhao.3035.8.20131214.

参考文献

［1］WANG Jianmin,REN Genquan,ZHANG Li,et al.Maintenance repair and overhaul/operations support technology［J］.Computer Integrated Manufacturing Systems,2010,16(10):2017-2025(in Chinese).［王建民,任艮全,张力,等.MRO支持技术研究［J］.计算机集成制造系统,2010,16(10):2017-2025.］
[2]ZHANG Yansong,ZHANG Yu,HUANG Wei,et al.Distributed aggregate functions enabled parallel main-memory OLAP query processing technique［J］.Journal of Software,2009,20(S1):165-175.
[3]DEAN J,GHEMAWAT S.MapReduce:simplified data processing on large clusters［J］.Communications of the ACM,2008,51(1):107-113.
[4]LAKSHMAN A,MALIK P.Cassandra:a decentralized structured storage system［J］.ACM SIGOPS Operating Systems Review,2010,44(2):35-40.
[5]ZHU Y,DU N, TIAN H,et al.LaUD-MS:an extensible system for unstructured data management［C］//Proceedings of Web Conference (APWEB)2010 12th International Asia-Pacific.Washington,D.C.,USA:IEEE,2010:435-440.
[6]CHANG F,DEAN J,GHEMAWAT S,et al.Bigtable:a distributed storage system for structured data［J］.ACM Transactions on Computer Systems (TOCS),2008,26(2):4.
[7]NIKOLOV P,PIERRE G.Aggregate queries in NoSQL cloud data stores［D］.Amsterdam,the Netherlands:VU University Amsterdam,2011.
[8]HAN Yi,HAN Weihong,YANG Shuqiang,et al.The study and implementation of an effective massive data storage technique to OLAP［J］.Microelectronic & Computer,2006,23(1):154-156(in Chinese).［韩毅,韩伟红,杨树强,等.面向 OLAP 的高效海量数据存储技术研究与实现［J］.微电子学与计算机,2006,23(1):154-156.］
[9]RAMESH B,KRAUS T B,WALTER T A.Optimization of SQL queries involving aggregate expressions using a plurality of local and global aggregation operations:USA Patent 5,884,299［P］.1999-03-16.

[1]	娄建楼,李燕,王琦,孙博,贾俊奇. 基于改进堆栈降噪自编码器的锅炉设备在线监测数据清洗方法[J]. 计算机集成制造系统, 2019, 25(12): 3181-3190.
[2]	李玲,刘敏,吴鸣. 基于复合维修物料清单的设备运行状态管理模型[J]. 计算机集成制造系统, 2016, 22(第6期): 1530-1541.
[3]	李玲,刘敏,成国庆,刘清. E-MRO模式下基于模糊随机规划的共赢服务策略[J]. 计算机集成制造系统, 2016, 22(第1期): 70-87.
[4]	钟雨,黄向东,刘丹,黄宇霞,田炜,王建民. 大规模装备监测数据的NoSQL存储方案[J]. , 2013, 19(12): 0-0.
[5]	曾令男,丁建伟,赵炯,张力,刘英博. 基于互信息的复杂装备高维状态监测数据相关性发现与建模[J]. , 2013, 19(12): 0-0.
[6]	郭建文,孙振忠,汤泓,黄汇江,张智聪. 设备e-维护知识网络模型[J]. , 2013, 19(12): 0-0.
[7]	岳挺,夏开渝,刘英博,张力. 一种风电产品的MRO支持系统解决方案[J]. , 2013, 19(12): 0-0.
[8]	张玮,杨善林,刘婷婷. 基于威布尔分布的MRO企业估算零部件维修更换率的方法[J]. 计算机集成制造系统, 2013, 19(09): 2237-2243.
[9]	郑小强,刘敏,孔繁荣,严隽薇. 基于云遗传算法的MRO服务调度[J]. 计算机集成制造系统, 2013, 19(09): 2348-2354.
[10]	雷达,钟诗胜. 面向MRO的民航发动机拆发日期预测系统[J]. 计算机集成制造系统, 2013, 19(07 ): 1715-1720.
[11]	李楠，刘敏，严隽薇. 面向钢铁连铸设备维护维修的工业物联网框架[J]. , 2011, 17(02): 0-0.
[12]	尹九波，严隽薇，刘敏. 面向钢铁连铸设备维护维修和大修的智能点检系统[J]. , 2010, 16(12): 0-0.
[13]	王建民，任艮全，张力，刘英博，莫欣农. MRO支持技术研究[J]. , 2010, 16(10): 0-0.
[14]	程曜安，张力，刘英博，莫欣农. 大型复杂装备MRO系统解决方案[J]. , 2010, 16(10): 0-0.
[15]	赵振，严隽薇，刘敏+，刘钢. 面向钢铁连铸设备MRO的企业语义集成及业务协同平台[J]. , 2010, 16(10): 0-0.