Lesson 3: lesson3.pdf
Video: https://www.youtube.com/watch?v=WeHM2xpYQpw&list=PLJV_el3uVTsMhtt7_Y6sgTHGHp1Vb2P2J&index=3&ab_channel=Hung-yiLee
Framework of ML
Training process
General Guidance
The first thing to check → Training data loss
Model Bias ⇒ 大海裡沒有針
Solution: redesign your model
Optimization ⇒ 大海裡有針找不到
Your optimization method is not good
Model Bias or Optimization
Which one???
Optimization issue
Take a deeper network for example.
Why 56-layer is worse than 20-layer ⇒ Optimization method is bad.
Solution:
- Gaining the insights from comparison
- Start from shallower networks (or other models), which are easier to optimize.
- If deeper networks do not obtain smaller losses on training data, then there is an optimization issue.
Loss on training data is small bit large on test data
Overfitting — an extreme example
Solution 1: more training data and data augmentation
Solution 2: constrained model
Bias-Complexity Trade-off
How to select a model
Why not use the testing set to select a model? ⇒ Model can cheat
Solution 1: Cross Validation
Train/Test: 90/10 or 80/20
Solution 2: How to split a training set? ⇒ N-fold Cross Validation
Mismatch
Training data and testing data have different distribution
⇒ Add complexity to the training set.
