The impact of internal noise on the performance and training of hardware artificial neural networks. Prospects for hardware spiking networks
Nadezhda I. Semenova1, Ivan D. Kolesnikov1, Daniil A. Maksimov1; 1Saratov State University, Saratov, Russia
Abstract
Over the past few years, artificial neural networks (ANN) have found their application in solving many problems. In terms of computational resources, ANN modeling is a very resource-intensive task. Despite the existence of powerful computing clusters with the ability to parallelize computations, modeling a neural network on digital equipment faces problems with network scaling, the speed of receiving or processing information, and energy efficiency. In recent years, more and more researchers in the field of neural networks are interested in creating hardware networks in which neurons and the connections between them represent a physical device capable of learning and solving problems. However, experimental and hardware setups always contain noise of various natures, so studying their influence is an urgent problem, the solution of which will help improve the efficiency of training in the presence of noise. Studying the impact of various types of noise in machine learning is aimed at determining the properties of noise effects that can accumulate in a neural network, or, conversely, be suppressed by the network itself.
Here, we mainly show how internal noise during neural network training affects the final performance of feedforward neural networks (FNNs) and echo state neural networks (ESNs). The noise types considered in this paper were initially inspired by real-world optical neural network implementations. However, these types have been generalized to improve the applicability of our results more generally. The noise types considered include additive and multiplicative noise, which depend on how the noise affects each individual neuron, and correlated and uncorrelated noise, which refers to the effect of noise on groups of neurons (such as the hidden layer of an FNN or the reservoir of an ESN).
Speaker
Nadezhda Semenova
Saratov State University
Russia
Discussion
Ask question
