文献类型：期刊文献

中文题名：面向外汇市场监测的分布式计算框架设计

英文题名：Design of distributed computing framework for foreign exchange market monitoring

作者：程文亮[1];王志宏[2];周虞[1];过弋[2,3,4];赵俊锋[1]

机构：[1]中汇信息技术(上海)有限公司开发二部,上海201203;[2]华东理工大学信息科学与工程学院,上海200237;[3]大数据流通与交易技术国家工程实验室(商业智能与可视化研究中心),上海200237;[4]上海大数据与互联网受众工程技术研究中心,上海200072

年份：2020

卷号：40

期号：1

起止页码：173

中文期刊名：计算机应用

外文期刊名：journal of Computer Applications

收录：CSTPCD;;北大核心:【北大核心2017】；CSCD:【CSCD_E2019_2020】；

基金：国家重点研发计划项目(2018YFC0807105);上海市科学技术委员会科研计划项目(17DZ1101003,18511106602,18DZ2252300)~~

语种：中文

中文关键词：外汇市场;市场监测;Spark;有限无环图;资源分配

外文关键词：foreign exchange market;market monitoring;Spark;Directed Acyclic Graph (DAG);resource allocation

摘要：针对金融外汇市场监测指标计算复杂度高、完备性强、效率低等问题,基于Spark大数据架构提出了一种新的面向外汇市场监测的分布式计算框架。首先,对外汇市场监测的业务特性和现有技术框架进行了分析总结;然后,综合考虑了外汇单市场多指标和多市场多指标并行计算的业务特性;最后,基于Spark的有向无环图(DAG)作业调度机制和YARN的资源调度池隔离机制,分别提出了外汇市场级的有向无环图(M-DAG)模型和市场级资源分配策略--M-YARN。实验结果表明,所提面向外汇市场监测的分布式计算框架相对于传统技术框架在性能上提高了80%以上,可以有效保证大数据背景下外汇市场监测指标计算的完备性、精准性和时效性。
In order to solve the index calculation problems of high complexity, strong completeness and low efficiency in the filed of financial foreign exchange market monitoring, a novel distributed computing framework for foreign exchange market monitoring based on Spark big data structure was proposed. Firstly, the business characteristics and existing technology framework for foreign exchange market monitoring were analyzed and summarized. Secondly, the foreign exchange business features of single-market multi-indicator and multi-market multi-indicator were considered. Finally, based on Spark’s Directed Acyclic Graph(DAG) job scheduling mechanism and resource scheduling pool isolation mechanism of YARN(Yet Another Recourse Negotiator), the Market-level DAG(M-DAG) model and the market-level resource allocation strategy named M-YARN(Market-level YARN) model were proposed, respectively. The experimental results show that, the performance of the proposed distributed computing framework for foreign exchange market monitoring improves the performance by more than 80% compared to the traditional technology framework, and can effectively guarantee the completeness, accuracy and timeliness of foreign exchange market monitoring indicator calculation under the background of big data.