APS360 — Applied Fundamentals of Deep Learning

Course introduces deep learning from an applied perspective in Python with PyTorch. Followed by ECE421 — Introduction to Machine Learning.

Recommended textbooks:

• Dive into Deep Learning, by Zhang, et al.
• Deep Learning, by Goodfellow, et al.

Concepts covered

Neural networks

• Neuron
  • Weights and biases
• Activation function
  • Linear activation function
  • Unit step function (and sign function)
  • Sigmoid
  • ReLU
  • Softmax
    • Temperature scaling
• Neural network layer
  • Decision boundary
• Neural network architectures
  • Feed-forward network (MLP)
  • Fully-connected network
  • Residual network
  • Convolutional neural network (CNN)
    • Convolution
    • Pooling
    • Existing architectures
      • LeNet
      • AlexNet
      • GoogLeNet
    • Transposed convolution
  • Autoencoder
    • Variational autoencoder (VAE)
    • Convolutional autoencoder
  • Recurrent neural network (RNN)
    • Long short-term memory (LSTM)
    • Gated recurrent unit (GRU)
    • Sampling strategies
      • Greedy search
      • Beam search
      • Temperature scaling
  • Graph neural network (GNN)
    • Deep set
    • Graph convolutional network (GCN)
    • Graph attention network (GAT)
• Model performance metrics
  • Error function
    • Mean-squared error
    • Cross entropy (CE) and binary cross entropy (BCE)
    • Negative log likelihood (NLL)
  • Classification metrics
    • Accuracy, precision, recall, F1-score, support
• Gradient descent
  • Vanishing and exploding gradient problem
  • Backpropagation
  • Optimiser
    • Stochastic gradient descent (SGD)
    • Adaptive moment estimation (Adam)
  • Learning rate
• Data processing
  • One-hot encoding
  • Dataset (splitting)
  • Data augmentation
  • Regularisation
    • Weight decay
    • Neuron dropout
• Batch normalisation
  • Layer normalisation
• Transfer learning

Artificial intelligence

• Symbolic artificial intelligence
• Machine learning
  • Types of ML approaches
    • Supervised learning
    • Unsupervised learning
    • Self-supervised learning
    • Contrastive learning
    • Deep learning
  • Types of ML problems
    • Regression
    • Classification
• Generative AI
  • Variational autoencoders (VAE)
  • KL divergence
  • Generative adversarial network (GAN)
    • CycleGAN
• Computer vision
  • Histogram of oriented gradients
  • Convolution image processing
• Natural language processing
  • Vector embedding
    • Approaches
      • word2vec
      • GloVe
    • Models
      • Skipgram
      • Continuous bag of words
  • n-gram
  • Sentiment analysis
• Transformer
  • Attention mechanism
• Limits of deep learning
  • Explainability
  • Adversarial attacks
  • Machine learning ethics (racial, gender discrimination)

Tools

• Development environments
  • Colab
  • Jupyter
  • PyCharm
• Python libraries
  • Numerical computing
    • NumPy
    • pandas
  • Machine learning
    • scikit-learn
    • PyTorch
      • torchtext
      • PyTorch Geometric
  • Visualisation tools
    • Matplotlib
    • t-SNE
• LaTeX