Skip to main content

3. History of Deep Learning, from Biology to Programming

3. History of Deep Learning, from Biology to Programming


3.1. How did everything started?


I am not going to include here a deep explanation(redundancies apart (; ) of the history of Deep Learning; but if you are interested, some useful resources I have found are:

However, I am going to explain the evolution from the Biological origin (sorry I am a Biomedical Engineering, I had to include my “bio-” prefix somewhere!)


3.2. Biological Neural Networks




3.3. The Basis of Biological Neural Networks: The Perceptron


Perceptron


The psychologist Frank Rosenblatt’s  conceived the Perceptron from the idea of a neuron. It was defined as a simplified mathematical model of how the neurons operate:
- It takes a set of binary inputs (nearby neurons)
- Each input is multiplied by a continuous valued weight (the synapse strength to each nearby neuron)
- It thresholds the sum of these weighted inputs to output a 1 if the sum is big, 0 otherwise (like the way neurons fire or not)

The biological aspiration can also be shown in the neuron model of Mcculoch-Pitts, who sums binary inputs and will output 1 if a certain threshold is exceeded, or a 0 if not.

Perceptron 2

Here the concept of Activation Function is introduced, known as the non-linear function applied to the weighted sum to produce the output of the artificial neuron.

Donald Hebb then stated: “When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased.”

This was not exactly followed by Perceptrons, but they could “learn” a function in which the weights would be modified after each training set valuation.

But a single Perceptron can only learn to output a 1 or a 0, thus multiple Perceptrons are needed in a layer to form neural networks that allow working with classification tasks; e.g. identifying the letters and digits in human handwriting.


Neural net with an output layer.



3.4. How hidden layers appeared



Hidden layers



Why hidden layers? They can find features within the data and allow following layers to operate on them (rather than on the noisy and large raw data).

When applying some calculus, we can assign an error made by “each output layer” and split the blame between it and the former hidden layer. This backwards process is known as “error backpropagation”, and it allows us to determine the difference in the net error when changing some weights and using an optimization technique. The objective is therefore to minimize this error.


Backprop



Comments

Post a Comment

Popular posts from this blog

4. Understand how a Neural Network works

4. Understand how a Neural Network works 4.1. Graphical Understanding In order to start in this world, it is important first to have a visual perception of what we are going to deal with, and understand the basics of how Neural Networks work, in the raw form. Image Source A Neural Network (NN) is nothing else than a net of perceptrons that are linked so that input fires another network that produces an output.  Of course, a NN has assigned values that allow to make further calculations and learning.   Image source So far the two images above represent a simple NN with 1 input, 1 output and 1 hidden layer. The hidden layer is said to be dense (each neuron in a layer x is connected to all neurons in the layer x-1 and all the neurons in the layer x+1). Depending on how the neurons are organized and how the connections are made, we can find many different types of NN:   Types of NN (image source and explanation) 4.2. Simple Neural
Post a Comment
Read more

Glossary of terms

  Image Source Machine learning (ML) : Subfield of Artificial Intelligence (AI) in the field of computer science focused on the development of intelligent systems, capable of "learning" (i.e., progressively improve performance on a specific task) with data , without being explicitly programmed. Ref Chronology : Build with GoogleCloud - Chronology tree for ML Perceptron : algorithm for supervised learning of binary classifiers (functions that can decide whether an input, represented by a vector of numbers, belongs to some specific class or not → classification in two parts). “It is a single layer NN”, thus in order to learn how a NN works, we should learn how a perceptron works. Weights : show the strength of the particular node Bias value : allows to shift the activation function Activation function : map input between required values like (0, 1) or (-1, 1).      Fig : Perceptron Deep learning : (also known as deep stru
Post a Comment
Read more

1. Definition

  1. Definition 1.1. What is Deep Learning?   “Deep learning is the state-of-the-art approach across many domains, including object recognition and identification, text understating and translation, question answering, and more. In addition, it is expected to play a key role in many new usages deemed almost impossible before, such as fully autonomous driving”, Ronny Ronen It looks like a lot of fun, right? I am sure you already know what you need Deep Learning for, which I bet is something fascinating. So without making it longer… Let’s start diving!! I will try to make this manual as concise and self-explanatory as possible. I still lack of experience so I apologize in advance if there is anything that is not clear or does not work. If you let me know I’ll do my best to improve the explanation or fix the problems as soon as I can. By the way, in this handeable manual we can find a glossary of terms in the end. I will try to set in italics (or other marking?) everytime a word a
Post a Comment
Read more