Neural Networks – Introduction

Hi fellows, after some time, I’ve get all things running now so I can start things out. The first topic I will present is a series of articles about neural networks. I will not focus on a serious posting about it, I will try to keep as simple as possible. Also, I will draw some useful pictures or show myself thoughts on pictures to make it more comprehensive. Now, let’s start with an introduction on neural networks, showing how they work.

First, I need you to remember how our nervous system works, especially our brain. We have a special cell there, the neuron, and we have a special process which makes the neurons communicate each other, the synapse. I will explain them here, so, don’t be afraid if you don’t remember.

Just think with me about a neuron, maybe you remember how is the form of a neuron, by I will imagine and you can see my imagination on picture above:



All right, this is a neuron! Here we have the axon, the soma and the dendrites. The neuron works receiving an electric impulse on dendrites terminations, this impulse travels to soma which kind of decides if this impulse must be sent to axon or not by generating an action potential. The soma has an activation threshold potential which the impulse must break in order to be transmitted to axon terminations. If the impulse has a potential high enough to generate an action potential, it can go forward. Simple, right? Well, this process is not that simple, I know. It is far more complicated, especially when we talk about action potentials and activation threshold potentials, but it is a good way to explain how the information is transmitted. The essential here is the electric impulse part. In our brain, all the data is transmitted in form of electricity. But how we can decode it to our digital world?

Let’s think a little bit. In our electric impulse, we have a potential, a measure of voltage right? But we don’t have data. At neurons, this process is sort of made by neurotransmitters, transfering those potentials through dendrites and axons terminations during synapses, I will talk about this process later. Now we need to work with a model to match the neuron. We cannot process the data itself, so that is the idea, we will learn with the data. We will not change the input data, but we will learn with it. The neuron act in the same way by using the action potentials. The data is the stimuli from outside, this is translated as a electric potential. We can do the same defining a weight for each input. A weight is a number which means something regarding the data input. This weight can be modified as long the input goes forward the artificial neuron. This weight can also be transmitted through connected neurons and changed by them, which we will see soon. As well in neuron, the artificial neuron also needs a way to sum up those weights in order to check if this sum can be transmitted out or not, so we define two functions for it. I think I confuse you now, saying a lot of things at the same time, right? Above is a picture of what I said:

artificial neuron


On picture above, you can see a vector x (x0,x1,..) and associated to it a vector w (w0,w1,..) . The vector x is our input and vector w is the input weights. We can start with some pre-defined weights or random weights here. In our artificial neuron, the first thing we have is our sum function which will apply a sum on vector w.x (we need to multiply the two vector in order to weight vector reflect input data behavior, I will talk more about this later), giving a number W, which is transferred to our activation function or thresholding function k. The output of neuron is k(W). It is common the activation function returns a binary output (0 or 1) but it is not the rule.

But a single neuron will not do much work for us, neither in a biological neural system. In our brain we have a vast network of connected networks of neurons. Each of them transferring information and making things working in our body. Across each neuron we have a process called synapse.


NN corrected


In a neuron connection we have the following: neuron dendrites linked on other neurons axons terminations. Between this link, we do not have physical contact between structures. Between them, works the synapse process. When the action potential comes out from axon, substances called neurotransmitters acting between the terminations in order to transfer the potential from axon to dendrites termination, initiating the neuron processing on other side. There is a very complex mechanic on neurotransmitter action on potentials but I will simplify stuffs here, we will only need to know about how the potential is transferred. For our artificial neural network, we can simplify and just connect the output of a neuron as the input of a new neuron. Now we have our “neurotransmitter process”, we can think about a neurotransmitter in our brain as a kind of modifier of the potential data. We do the same thing in our artificial neurons. As soon the input come into a neuron, we multiply it with the weights on neuron (as we did above with w.x), giving us a new weight vector with information of neuron weights but with input characteristics. This process permits our artificial neural network to learn and pass this learning across neurons on network, as our brain does.


Above a simple neural network with a hidden layer. I will explain the layers of a neural network on next article and show how to implement a simple neural network, the perceptron network. I will show some implementation in python as well letting you see the network working!


See you layer!