Accelerating Map-Reduce using Distributed Cache as Middleware with Partial Pre-shuffling on Hadoop Cluster and Securing Memcached Server - Accelerated Map-Reduce Implementation on Hadoop Cluster

  Chandrashekhar Singh,  Ayushi Srivastava

Map-Reduce is a widely-used model for data parallel applications enabling easy development of scalable programs on clusters of commodity machine. Advancements in disk capacity have greatly surpassed those in disk access time and bandwidth. As a result, disk based systems are finding it increasingly difficult to cater to the demands of a cluster-based system. A cache memory has faster data access rate than normal disk. In this paper, we have implemented a method to improve the performance of map-reduce by using distributed memory cache as middleware between the map and reduce phase with highly secure memcached server. Moreover, it provides partial combining of data along with the Map task itself, so called partial pre-shuffling. The project also provides data assurance of intermediate key-value pairs via a web interface. The results of the overall setup were promising over a Hadoop cluster.

Keywords- Distributed File System, Hadoop, Map-Reduce Memcached, High throughput, High speed data access, Cluster-based computing, Middleware.

Unique Identification Number - IJEDR1404005
Page Number(s) - 3390-3393
Pubished in - Volume 2 | Issue 4 | Dec 2014
DOI (Digital Object Identifier) -   
Publisher - IJEDR (ISSN - 2321-9939)

  Chandrashekhar Singh,  Ayushi Srivastava,   "Accelerating Map-Reduce using Distributed Cache as Middleware with Partial Pre-shuffling on Hadoop Cluster and Securing Memcached Server - Accelerated Map-Reduce Implementation on Hadoop Cluster", International Journal of Engineering Development and Research (IJEDR), ISSN:2321-9939, Volume.2, Issue 4, pp.3390-3393, Dec 2014, Available at :http://www.ijedr.org/papers/IJEDR1404005.pdf