Formal neuron of a new type
Omega Server
Spread the love

Vladimir Proseanic, Boris Zlotin, Anatol Hin

In 1943 Warren S. McCulloch and Walter Pitts [1] for the first time in history mathematically described the functioning of the thinking apparatus of humans and higher animals, introducing the concept of a formal neuron – as they called a simple signal processing device that has some properties of real brain cells – biological neurons. They also introduced the concept of an artificial neural network, that is, a network of formal neurons built according to the same rules as the neural networks of a real brain.

Formal neuron and artificial neural network have become possible to implement using electronic or electrical elements. In the fifties, on the basis of the rapidly developing computer technologies, the first artificial neural networks were built, confirming the correctness   of the McCulloch and Pitts’s main ideas about the possibility of modeling the basic elements of thinking. It has also been shown that such networks can be trained and that, based on this training, they can perform the following processes and operations:

Initially, the rapid creation of full-fledged artificial intelligences based on neural networks was expected, but, unfortunately, these hopes turned out to be unrealistic. Why did it not happen?

Even today, after more than 70 years of development, the full-fledged artificial intelligence is far from creation. This is due to two factors inhibiting development:

Experts from Progress, Inc., Detroit, discovered the reasons for such a strong difference between artificial and biological networks. In the description of the formal neuron by McCulloch and Pitts, an error was made due to the lack of some important data on the functioning of biological neurons at that time.

A biological neuron was considered a relatively simple device that summarizes signals coming from other neurons through contact units called synapses, and in which signals are transmitted through the emission of a special chemical called a neurotransmitter (Dale’s doctrine). Each neuron has tens of thousands of synapses and, as it was then believed, one type of mediator works in each synapse, and the resistance of the synapse to signal passage (its biological mechanism is very complex and not very important for us) is usually called synaptic weight.

The formal neuron of McCulloch and Pitts repeated the structure of a biological neuron and looked like a simple circuit – an electronic or electrical adder that receives signals from a variety of variable resistances that play the role of synaptic weights.

A neural network of this type reduces training to the selection of resistance values ​​(weights) of all synaptic weights, which requires thousands and sometimes millions of training epochs and a huge amount of computation in each of these epochs.  The network “intelligence” grows approximately linearly with an increase in the number of neurons and the volume of training, and the total amount of computations and, accordingly, the time and energy spent, grow approximately in proportion to the square of the growth in the number of neurons and the volume of system training. Thus, networks designed to perform more or less intelligent work have to be trained for months and years.

There are many ways to speed up training, for example, backpropagation systems, deep learning methods on networks with many neural layers, the use of ultra-precise networks that train the network by sections, etc. This makes training many times faster, but does not remove the problem of the exponential increase in training time.

Doyle’s doctrine was experimentally refuted by biologists back in the seventies of the 20th century, when it was discovered that several different neurotransmitters can work simultaneously in a biological synapse, and, therefore, the formal neuron of McCulloch and Pitts describes operation of a biological neuron incorrectly. However, all traditional and modern artificial neural networks were and are still being built on the same formal neurons.

Progress specialists have developed a new formal neuron (progressive formal neuron or p-neuron), which includes a set of corrective weights that act as various neurotransmitters. P-neuron is much closer to a biological neuron in design and principle of action. The figure below shows a typical p-neuron and a network built on such neurons. In the p-neuron, the signal from the input device goes to the signal analysis and distribution unit (distributor), which, having estimated the value of the signal, assigns it to one of the intervals and, in accordance with this, assigns a corrective weight corresponding to this signal. The figure shows that a signal with a value corresponding to the interval of values ​​3 selects the corrective weight d3.

The most important result of the application of a new type of formal neuron was a radical change in the network training, more consistent with the training processes in biological neurons, and therefore requiring billions of times less computations. This also provides the ability to train and retrain network fast (e.g. in real time).

A new type of formal neurons can be implemented in computers using conventional computer facilities, such as a CPU, as well as graphics cards, specialized or programmable microchips, analog devices, etc. P-neurons can provide significant improvements to known types of neural networks, for example, single- and multilayer perceptrons, cognitrons, Kohonen, Hopfield, Boltzmann networks, deep learning networks, convolutional networks, etc.

The so-called Progress networks (p-networks), created by Progress, Inc., specifically for p-neurons’ application, can also be used. P-network can also be analogous to convolutional networks; in particular, as a result of folding a set of single-layer perceptrons, each of which is trained to recognize one image. This allows additional training of the network, which is practically reduced to adding one more “monotonous” perceptron to the finished convolution.

Due to the new training process, which reduces the number of necessary mathematical operations by many orders of magnitude, it becomes possible to implement full-fledged neural networks on computers with low computing power, for example, on laptops, tablets, phones, cheap controllers, etc.

Artificial neural networks built on new formal neurons (p-neurons) have some additional capabilities:

The most important thing, in our opinion, is the possibility of creating, on the basis of p-networks, a full-fledged specialized artificial intelligence of a wide profile, capable of constantly learning and working in real time.

The basics of the theory and design of p-neurons and p-networks are described in articles [2], [3] and in patents [5].

References:

  1. Warren S. McCulloch & Walter Pitts. «A logical calculus of the ideas immanent in nervous activity» 1943
  2. D. Pescianschi, Main Principles of the General Theory of Neural Network with Internal Feedback, CSREA Press., Las Vegas, US 2015
  3. D. Pescianschi, A. Boudichevskaia, B. Zlotin, and V. Proseanic, Analog and Digital Modeling of a Scalable Neural Network, CSREA Press., Las Vegas, US 2015
  4. A new era in development of information technology. Creatime Project, site
  5. Patents: US Patent No. 9390373, US Patent No. 9619749, US Patent No. 10423694,  Japan Patent No. 6382354, China Patent No. ZL201580012022.2, Mexico Patent No. MX357374B,  Taiwan Patent No. I655587,  Israel Patent No. 247533,  Hong Kong Patent No. HK1227499,  Singapore Patent No. 11201608265X,  Eurasian Patent No. 035114