Auto Determination of K in KMEANS with MAP-REDUCE for Numerical and Text Datasets

Abstract

Data mining is the process of automatically discovering useful information in large datasets. Clustering analysis is a very important branch in data mining. Cluster analysis based on the data objects and their relationships and grouping of data objects. Clustering very large datasets is a challenging problem for data mining and processing. Map Reduce is considered as a powerful programming framework, which significantly reduces executing time by dividing a job into several tasks, and executes them in a distributed environment. K-Means, which is one of the most used clustering methods, and K-Means based on Map Reduce is considered as an advanced solution for very large dataset clustering. However, the executing time is still an obstacle due to the increasing number of iterations when there is an increase of dataset size and number of clusters. The traditional k-means is computationally expensive, sensitive to outliers and has an unstable result hence its inefficiency when dealing with very large datasets. Solving these issues is the subject of much recent research work. In this paper, we propose an Auto determination of K in KMEANS with MAP-REDUCE for numerical and text datasets in order to adapt it to handle large-scale datasets by reducing its execution time. In addition, we proposed algorithms to find the initial centroids automatically and cluster are formed on both numerical and text both datasets.