Fully-connected networks are a type of neural network where each neuron in each layer is connected with each other. In other words, the output of a neuron in a given layer is connected to every neuron in the next layer.
For example:1
Improvements
Problems with fully-connected networks:
- Computational complexity grows with variations in the input. This means things are harder to train and there needs to be more data to generalise the model.
- Bad inductive bias: for vision problems, it ignores the geometry of the image data (i.e., different dimensions can screw up the image). It throws out prior understanding/knowledge of the image.
- Not flexible: different image sizes require different models.
This motivates the idea of the convolutional neural network, which is able to rectify these problems.
In code
In PyTorch, the torch.nn.Linear(in: int, out: int) function defines a fully-connected (dense) layer. The below are the linear transformations used to link each layer:
self.layer12 = nn.Linear(4, 5)
self.layer23 = nn.Linear(5, 3)After applying each linear transformation, we need to apply an activation function, i.e.,:
activation1 = torch.nn.functional.relu(self.layer12(input))Footnotes
-
From Dive Into Deep Learning, by Zhang, Lipton, Li, and Smola. ↩