DLFA: Cohort I Deep Learning and its Applications

Module 3: Deep Learning and its Applications

Deep Learning Architectures

• August 28, 2021
  
1. Gradient descent and its variants, 
2. Log likelihood Interpretation of Linear Regression, 
3. Logistic Regression - Binary classification model, 
4. Sigmoid or logistic function, Loss function, Binary cross entropy, 
5. Logistic Regression - Multiclass classification
6. Activation Function - Softmax Function
7. Feedforward Neural Network
8. Rectified Linear Units(ReLU)
9. Leaky ReLU
10. Hyperbolic tangent function
11. Chain Rule of Calculus
12. Backpropagation
• September 04, 2021
1. Recap of Backpropagation, 
2. Relu and Softmax Activation Function, 
3. L2 Regularization, 
4. Early stopping, 
5. Dropouts
6. Validation Set and Validation Error
7. Classical Features and Deep Features
8. Batch Normalization
9. Stochastic Gradient Descent
10. Momentum based optimization
• September 12, 2021
• September 12, 2021
1. Optimization - Momentum, Adam, Adagrad, RMSprop;
2. Batch Normalization;
3. 1D Convolution;
4. 2D Convolution;
5. Convolutional Layer - Filter, Stride
6. Zero Padding
7. Max Pooling
• September 19, 2021
1. Recurrent Neural Networks (RNN);
2. First Order Recurrence - Hidden Layer;
3. Why do need Recurrent Models;
4. Error Back Propagation;
5. Backpropagation through time
6. Long term dependency issue - Vanishing Gradients
7. Tanh Activation Function Long Short Term Menory (LSTM)
8. Recurrent networks - Single Input Multiple Outputs
9. Bi-directional Netorks
10. Sequence to Sequence Mapping Networks
• September 25, 2021
1. RNN;
2. Attention Encoder Decoder Model
3. Attention in LSTM networks;
4. Natural Language Processing – Representation of Text
5. Word2vec models as text representations - continuous bag-of-words and skip-gram model
6. Word2vec visualization
7. Deep Neural Network-Intution
8. CNN for MNIST
9. Understanding CNN's using CNN's
10. Problem of Underfit and Overfit
11. Regularization in Deep Network
12. Batch Normalization"
• October 09, 2021
1. ASR – Statistical Speech Recognition
2. Probabilistic Approach- Statistical Sequence Recognition
3. Acoustic Model – State based modeling approach – HMM(Hidden markov model);
4. HMM decoding
5. Word Error Rate/Character Error
6. ASR – DNN/HMM Hybrid Model
7. ASR – TDNN(Time Delay Neural Network)
8. RNN
9. Training a RNN language model
10. Evaluating language model
11. ASR -End to end approach
12. NMT(Neural Machine Translation)
13. Greedy decoding
14. Seq-to-seq with attention
15. Self attention
16. The transformer encoder-decoder
17. CTC(Connectionist Temporal Classification)
18. Fully convolutional speech recognition
19. Speech transformer
20. Wav2vec
21. Autoregressive Predictive coding
• October 16, 2021
1. Concatenative TTS;
2. Concatenative Synthesis
3. Three stage pipeline
4. Vocoder
5. Source filter model
6. Maximum likelihood estimation of spectral model
7. Introduction of auditory frequency scale
8. Mel Generalized Ceptral Analysis Advanced Vocoder and excitation models
9. Speech Signal Processing Toolkit
10. MELP Style Mixed Excitation
11. Autoregressive Model - core idea
12. Autoregressive Model - wavenet
13. STRAIGHT
14. Excitation Signal Generation in STRAIGHT
15. Griffin Lim Algorithm
16. Naive Model
17. Non-autoregressive model – melGAN
18. Normalizing flow
19. Neural Source filter model
20. SFNet
21. Text Front End
22. Prosodic Production prediction
23. Pronunciation prediction
24. Style Information
25. Contextual Information
26. HMM Neural Network
27. Tacotran
28. Acoustic Model
29. Non-RNN Acoustic Model
30. Robust Acoustic Model
31. Non-autoregressive acoustic model
32. Accurate Alignment
33. End to end adversial text to speech
• October 23, 2021
1. CNN Architectures - Brief Overview;
2. AlexNet, VGG16, VGG19
3. ResNet,Self-Supervised Learning;
4. Contrastive Learning
5. Image Augmentations
6. A simple Framework for Contrastive Learning of Visual Representation
7. Contrastive Loss Function
8. Image Segmentation - UNet
9. Performance Metrics - Intersection over Union
10. DeepLab,Bilinear Interpolation
• October 30, 2021
1. Computer Vision and Semantic Gap
2. Conventional vs Deep Vision
3. AlexNet
4. FCNN
5. Object Detection (RCNNs)
6. Object Detection (Faster RCNN + ResNet-101)
7. Semantic Object Segmentation
8. RPN,VGG,GoogleNet,RCNN,YOLO
• November 06, 2021
1. GAN’s
2. Spatial Localization Detection
3. RCNN
4. Fast R CNN- Region Proposal Network
5. YOLO Detection as Regression
6. Adversial Robustness of Deep Models
• November 13, 2021
1. Image Captioning
2. Convolutional Feature Extractor
3. Model : Attention
4. Yolo V1, V2, V3
• November 27, 2021
1. Introduction to NLP and its applications
2. Sentence Classification
3. Textual Entailment
4. Question Answering
5. Self Supervised Learning
6. Text Preprocessing
7. Tokenization
8. Filtering Stop Words
9. Remove Punctuations
10. Stemming and Lemmatization
11. Additional Text Processing
12. Distributed Word Representations
13. Text Representation
14. BOW (Bag of words)
15. Cosine Similarity
16. Word2vec
17. Skip-gram model
18. GloVe
19. fasText
20. Sentence Representation
21. Deep Averaging Network
• December 04, 2021
1. Language Model
2. Statistical Language Model
3. N-gram Language Model
4. Sentence classification problem
5. Neural Machine Translation
6. Training on RNN Language Model
7. Generating Text with a RNN Language Model
8. Evaluating a Language Model
9. Bidirectional RNN
10. Embeddings from Language Model
11. Gated Recurrent Unit (GRU)
12. Convolutional Neural Netorks
13. CNN for Sentence Representation
14. Single Layer CNN for Sentence Classification
15. CNN for Machine Translation
16. Combination of CNN and RNN for paragraph Classification
17. CNN for Part-of-Speech Tagging
• December 11, 2021
1. Attention Mechanism
2. GRU Network
3. Data Fitting Problem
4. Bidirectional RNN
5. Attention mechanism for seq-to-seq problem
6. Self-attention
7. Transformers
8. Encoder
9. Multi head attention
10. Positional encodings
11. Transformer encoder for classification
• December 18, 2021
1. Backpropagation through time
2. Vanishing Gradient
3. GPT(Generative Pretrained Model)
4. Formatting inputs for fine-tuning task
5. English grammer correctioin using gpt 3
6. Bytepair tokenizer
7. Wordpiece tokenizer
8. BERT
9. BERT- Masked Language Model
10. BERT Variants
11. ine tuning bert on different task
• January 22, 2022
1. First Half: Techniques for efficient hardware inference
2. Separable Convolution
3. SparsityPruning Neural Networks
4. Recovery Accuracy
5. Pruning RNN and LSTM
6. Load balance aware pruning
7. Bitmask Compression
8. Number Representation
9. Quantization
10. AlexNet on ImageNet Classification
11. Trained Quantization
12. Huffman Coding
13. Ternary Net
14. Dataflows
15. Reduced Precision Networks
16. Non-linear quantization
17. Google TPU generation
18. Edge TPU
19. Machine Learning for IoT edge computing
20. Power Constraints
21. ML Hardware Tradeoffs
22. CPU and GPUF
23. PGA vs ASIC Chip 1(Analog ML)
24. VLSI Implementation: Weighted Average Circuit
25. Memristor
26. Microcontrollers
27. MCU(Cortex Series)
28. tinyML hardware
• January 30, 2022
1. Machine Learning on embedded system
2. Attributes
3. Framework
4. Design
5. MCU Hardware
6. ESP-EYE Dev kit
7. Arduino Nano 33 BLE
8. tinyML software frameworks
9. TensorflowLite Micro
10. TensorflowLite Micro NN Ops
11. TensorflowLite Conversion : Keras Model
12. TensorflowLite Model Optimization
13. Post Training Interger Quantization
14. Quantization Aware Training
15. Pruning
16. Weight Clustering
17. TFLM Structure
18. Installation
19. Hello World TensorflowLite Micro Screencast
20. Helloworld TFLM Components
• February 05, 2022
1. tinyML Deployment
2. Keyword spotting application
3. Audio sensor input
4. KWS dataset - Google speech command
5. Data Preprocessing - MFCC
6. Cochlea based filtering
7. tinyConv model
8. KWS components
9. Keyword spotting code demo
10. Custom model for TFLM framework
11. Visual fake words - Person detection application
12. Person detection metrics
13. Person detection model
14. person detection components
15. Person detection code demo
16. Sensor ecosystem
17. Gesture recognition demo