The Deep Learning revolution has disrupted the world of information technology and information economy. It has brought us driverless cars, Siri and Alexa, diagnosed cancer to name a few of the technologies and systems that it has facilitated. Deep learning belongs to the much broader family of machine learning. It is based on learning data interpretation instead of the conventional task-specific algorithms.
What’s unique about Deep learning is that the process can be compared to how babies get acquainted with the world. The deep learning models are associated with the information processing and communication patterns of neurons system. The word ‘deep’ comes from the multiple layers of processing units, each layer uses the output of the previous layer as input.
Rina Dechter was the one who coined the term Deep learning in 1986. Deep learning began gaining momentum in 2000’s but it was only in 2012 that the Deep learning revolution really began. It was in this year itself that a team being led by Dahl won ‘Merck Molecular Activity Challenge’. They had used the deep neural network to find the biomolecular target of one drug. In the year 2014, Hochreiter’s group used deep learning to find the toxic effects of chemicals in nutrients and drugs.
There was a surge of interest in deep learning. It went from being a purely technical field which interested few academics to being completely mainstream, so much so that papers on deep learning are being published in famous reputed journals such as Nature, etc. The breakthrough happened in 2012 when neural model accomplished the state of art in speech recognition when using deep nets.
But what happened in 2012 was the result of years of research and work. The basic idea of deep learning has been around since 1940’s when the concept of making machines that could learn like humans first arose. So even though the main concept behind deep learning has been in the minds of people for a really long time, its actual power couldn’t have been tested before really large databases were available and they were combined with GPU computing. This is why deep learning revolution began when it did, in the year 2012.
However, a significant event which deserves a mention here happened in 2010 when a large database known as Imagenet which had millions of labeled images was made by Fei-Fei Li’s group at Stanford. This was combined with the annual LSVRC, where contestants build computer vision models and then enter their predictions. After that, they would receive a score on the basis of accuracy. Initially, the error rates in the models were in the range of 28-26%. In 2012, the submission by Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton would have the error rate of 16%. Their model had combined many important parts which would go on to become the pillars of deep learning models.
Standford Vision Lab
It was only after this, that Image classification became a challenging task of creating descriptions for images, often as a combination of CNNs and LSTMs. Many researchers are of the opinion that the "deep learning revolution" was started because of the ImageNet victory, which has transformed the AI industry.
The most critical aspect was the use graphics processing units (GPUs) to train the model. The speedup caused because of GPUs meant that the training of larger models was now easier and quicker, which would, in turn, result in lower error rates. This was the moment which defined the renewed zeal into the AI industry and has convinced many the Great AI awakening is very near.
Some factors which played into the spearheading the deep learning revolution are:
Large datasets – As discussed, it was only the advent of large datasets which were labeled, that made us realize the true extent of the power of deep learning.
GPUs – The neural nets are floating point parallel calculations and GPUs are very quick and efficient at it. It was the change from CPU-based training to GPU-based training that has enabled in impressive speed ups for the models.
Better architectures - Resnets, inception modules, and Highway networks assisted in smooth flow and allowed people to change hyperparameters and increasing the depth and flexibility of the network.