An Improved ND-Adam Algorithm for Neural Networks to Reduce the Learning Error Rate

Abstract

RMSprop and Adam are two adaptive optimization approaches that have been shown to achieve better performance in several scenarios resulting from stochastic gradient descent (SGD). In a more recent study, summaries performed worse when comparing SGDs, which lie mostly in DNN (deep neural network) training. This paper identifies cases where Adam’s generalization is worse than SGD’s. And Adam’s version is inevitable to remove the simplification of the gap. NDAdam (Normalized direction preserving Adam) is proposed, which implements step size when updating weight vector and controls direction with high precision and advantages in improving generalization performance. The same logic is followed to increase the efficiency of summarizing classification tasks via softmax logits. We were able to develop gap in Adam and SGD and have the opportunity to explain need of the certain optimization approaches perform better than some existing algorithms.