Parallel computing for artificial neural network training

Abstract

The big-data is an oil of this century. A high amount of computational power is required
to get knowledge from data. Parallel and distributed computing is essential to
processing a large amount of data. Artificial Neural Networks (ANNs) need as much
as possible data to have high accuracy, whereas parallel processing can help us to save
time in ANNs training. In this paper, we have implemented exemplary parallelization
of neural network training by dint of Java and its native socket libraries. During the
experiments, we have noticed that Java implementation tends to have memory issues
when a large amount of training data sets are involved in training. We have remarked
that exemplary parallelization of a neural network training will not outperform drastically
when additional nodes are introduced into the system after a certain point. This
is widely due to network communication complexity in the system.