Institutions | About Us | Help | Gaeilge
rian logo

Go Back
BigDataNetSim: A Simulator for Data and Process Placement in Large Big Data Platforms
Batista de Almeida, Leandro; Cunha de Almeida, Eduardo; Murphy, John; De Grande, Robson E.; Ventresque, Anthony
The 2018 IEEE/ACM 22nd International Symposium on Distributed Simulation and Real Time Applications (DS-RT) Big Data platforms are convoluted distributed systems which commonly comprise skill- and labour-intensive solution development to treat inherent Big Data application challenges. Several tools have been proposed to help developers and engineers to overcome the involved complexities in coordinating the execution of plenty processes/threads on multiple machines. However, no work so far has been able to combine both an accurate representation of Big Data jobs and realistic modeling of the behaviour of Big Data platforms at scale, including networking elements and data and job placement. In this paper, we propose BigDataNetSim, the first simulator which models accurately all the main components of the data movements in Big Data platforms (e.g., HDFS, YARN/MapReduce, network topologies, switching/routing protocols) in a large scale system. BigDataNetSim can serve as a valuable tool for engineering Big Data solutions, which includes set-up of systems, prototyping of jobs, and improvement of components/algorithms for Big Data platforms. We also demonstrate that BigDataNetSim can simulate a real Hadoop cluster with a high degree of accuracy in terms of data and job placements, being able to scale up to very large systems.
Keyword(s): Big data; Hadoop; Simulation; Task analysis; Data models; Tools; Network topology; Protocols; YARN
Publication Date:
Type: Other
Peer-Reviewed: Unknown
Language(s): English
Institution: University College Dublin
Publisher(s): IEEE
First Indexed: 2019-05-24 06:15:31 Last Updated: 2019-05-24 06:15:31