Welcome to my very first blog post. This will be the beginning of a series of posts on some of the fundamental architectures in all of Deep Learning. This post will disseminate the now 22-year old paper by Yann LeCun et al. titled Gradient-Based Learning Applied to Document Recognition.

I’ll be focusing on the LeNet-5 architecture over Graph Transformer Networks(GTN), Space Displacement Neural Networks(SDNN) and some other things brought up in the 46-page paper because of their complexity and that LeNet-5 outperformed these approaches. (SDNNs are now a relic of the past).

LeNet-5 laid down the framework of today’s AI landscape. It is the first paper that applied the modern Convolutional Neural Network with gradient-based learning to a real-world application. Prior to LeNet-5, convolutional neural networks rarely had any applications. Instead, static hand-crafted features and Support Vector Machines(SVMs) were the norm.

Convolutional Neural Networks (referred to as CNNs or ConvNets) are a series of giant, mysterious blocks of matrix multiplications, non-linearity functions, and differential calculus and statistics. I will cover the individual sections of a neural network in future blog posts. I will, on the other hand, lay down a list of the essential blocks below.

  • Convolutions
  • Non-Linearities
  • Backpropagation
  • Objective/Loss Functions
  • Optimizer Functions
  • Layer Types (Pooling, Fully-Connected, Convolutional etc.)

While there are many other concepts that merit being in the list (like Batch Normalization for example), the 6 above are essential to all neural networks.

##LeNet-5## LeNet-5, as the name suggests, is a 5 layer convolutional neural network created by Yann LeCun, Léon Bottou, Yoshua Bengio and Patric Haffner. The model was first used to classify handwritten digits in the MNIST dataset. I will be recreating this model using PyTorch. All relevant scripts will be available on my GitHub repository.

####Libraries#### Before we get to the core implementation details, we need to import some libraries. The libraries are completely open-source. I will be using PyTorch in all of my work.

import torch import torch.nn as nn import numpy as np import matplotlib.pyplot as plt import torchvision

These are standard library imports required for all models written in PyTorch.

LeNet Architecture LeNet-5 in all its glory being used in digit recognition. Image taken from the original paper.