Dynamic Mapreduce for Job Workloads through Slot Configuration Technique

Authors(2) :-M. Priyanka, Dr. A. Subramanyam

The MapReduce is an open source Hadoop framework implemented for processing and producing distributed large Terabyte data on large clusters. Its primary duty is to minimize the completion time of large sets of MapReduce jobs. Hadoop Cluster only has predefined fixed slot configuration for cluster lifetime. This fixed slot configuration may produce long completion time (Makespan) and low system resource utilization. The current open source Hadoop allows only static slot configuration, like fixed numbers of map slots and reduce slots throughout the cluster lifetime. Such static configuration may lead to long completion length as well as low system resource utilizations. Propose new schemes which use slot ratio between map and reduce tasks as a tunable knob for minimizing the completion length (i.e., makespan) of a given set. By leveraging the workload information of recently completed jobs, schemes dynamically allocates resources (or slots) to map and reduce tasks.. Many scheduling methodologies are discussed that aim to improve execution performance as well as completion time goal.

Authors and Affiliations

M. Priyanka
M.Tech., (PG Scholar), Department of CSE, Annamacharya Institute of Technology & Sciences, Rajampet, Kadapa, Andhra Pradesh, India
Dr. A. Subramanyam
Professor, Department of CSE, Annamacharya Institute of Technology & Sciences, Rajampet, Kadapa, Andhra Pradesh, India

Map Reduce, Makespan, Workload, Dynamic Slot Allocation.

  1. http://hadoop.apache.org/docs/r1.2.1/fairscheduler.html
  2. http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop yarnsite/CapacityScheduler.html
  3. Ching-Chi Lin, Pangfeng Liu, and Jan-JanWu. Energy-aware virtual machine dynamic provision and scheduling for cloud,. In Cloud Computing (CLOUD), 2011 IEEE International Conference on, pages 736 –737, july 2011.
  4. Anton Beloglazov and Rajkumar Buyya. Energy efficient allocation of virtual machines in cloud data centers, In 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing, pages 577–578, 2010.
  5. Yibin Wei, Ling Tian , Research on cloud design resources scheduling based on Genetic Algorithm, 2012 International Conference on systems and informatics(ICSAI 2012)
  6. Chen, K. ; Powers, J. ; Guo, S. ; Tian, F. CRESP: Towards Optimal Resource Provisioning for MapReduce Computing in Public Clouds , IEEE Transactions on Parallel and Distributed Systems ,Volume: 25 , Issue: 6 Publication Year: 2014 , Page(s): 1403 – 1412.
  7. Xiaohong Zhang ; Yuhong Feng ; Shengzhong Feng ;Jianping Fan ; Zhong Ming An effective data locality aware task scheduling method for MapReduce framework in heterogeneous environments, 2011 International Conference on Cloud and Service Computing (CSC)Year: 2011 , Page(s): 235 - 242 8Sewoog Kim ; Dongwoo Kang ; Jongmoo Choi ; Junmo Kim Burstiness-aware I/O scheduler for MapReduce framework on virtualized environments , 2014 International Conference on Big Data and Smart Computing (BIGCOMP) Publication Year: 2014 , Page(s): 305 – 308.
  8. Hammoud, M. ; Rehman, M.S. ; Sakr, M.F. Center-ofGravity Reduce Task Scheduling to Lower MapReduce Network Traffic , 2012 IEEE 5th International Conference on Cloud Computing (CLOUD) Publication Year: 2012 , Page(s): 49 – 58.
  9. J. Polo, D. Carrera, Y. Becerra, J. Torres, E. Ayguad´e, M. Steinder, and I. Whalley. Performance-driven task co-scheduling for MapReduce environments. In 12th IEEE/IFIP Network Operations and Management Symposium. ACM, 2010.
  10. L. Phan, Z. Zhang, B. Loo, and I. Lee. Real-time MapReduce Scheduling. Tech. Report No. MS-CIS-10-32, UPenn, 2010.
  11. B. Palanisamy, A. Singh, L. Liu, and B. Jain. Purlieus: localityaware resource allocation for MapReduce in a cloud. In Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2011.
  12. Resource management with VMware DRS http://www.vmware.com/pdf/vmware_drs_wp.pdf.

Publication Details

Published in : Volume 2 | Issue 4 | July-August 2017
Date of Publication : 2017-08-31
License:  This work is licensed under a Creative Commons Attribution 4.0 International License.
Page(s) : 695-699
Manuscript Number : CSEIT1172488
Publisher : Technoscience Academy

ISSN : 2456-3307

Cite This Article :

M. Priyanka, Dr. A. Subramanyam, "Dynamic Mapreduce for Job Workloads through Slot Configuration Technique", International Journal of Scientific Research in Computer Science, Engineering and Information Technology (IJSRCSEIT), ISSN : 2456-3307, Volume 2, Issue 4, pp.695-699, July-August-2017. |          | BibTeX | RIS | CSV

Article Preview