Neural networks have many neurons that function as adjustable nodes, grouped together in layers. The number of layers (or the depth of the model) determines the number of decision boundaries our model has. Many problems require more than one decision boundary, which motivates the idea of multilayer NNs, with hidden layers in between the input and output layer. These introduce non-linearities into the model that allow it to separate data such that the final layer can use linear separation on the data.
PyTorch
In PyTorch, we can use a linear transformation that maps between one layer to the other with nn.Linear(in_neurons, out_neurons). This defines a fully-connected (dense) layer.
The first input determines how many input neurons we have. The second determines how many output neurons we have.
For this example:1
We would need to apply a linear transform between each layer as follows:
self.layer12 = nn.Linear(4, 5)
self.layer23 = nn.Linear(5, 3)After we apply each linear transformation, we can apply the corresponding activation function for the layer.
Note (especially for vision applications) that a 2D input, like an image, will have input neurons for each pixel.
Footnotes
-
From Dive Into Deep Learning by Zhang, Lipton, Li, and Smola. ↩