Plans for week 36, September 1-5

Dear all, welcome back to a new week. We hope you've had a nice and relaxing weekend. 

Here are the plans for the coming week. 

We will discuss in more detail the mathematics of ordinary least squares, Ridge regression and Lasso regression, introduced at the end of the lecture last week. This is will be the material for the first lecture on Monday. Thereafter (second lecture) we will start discussing the numerical solution of the optimization problem using gradient methods, or what are normally called gradient descent methods. 

Ordinary Least Squares and Ridge regression are methods where we have an analytical solution for the optimal parameters. For Lasso regression we need to solve the equations numerically. This is the standard situation in essentially all machine learning methods.  Introducing gradient methods, we will thus also introduce the recipe for solving the equation for the Lasso regression and all later methods.  These methods are based on the famous Newton-Raphson iterative method.

This week we will explore the simplest possible approach to gradient descent, where we introduce a new parameter called the learning rate (replaces the second derivative).  We will show how we can solve numerically ordinary least squares and Ridge regression and compare the numerical solution with the analytical solutions (which involve matrix inversion). 

The exercises this week deal with implementing the analytical solution of Ridge regression while next week we will implement our first variant of gradient descent for OLS, Ridge and Lasso. 

Next week will also discuss more advanced gradient descent methods. All these methods will be included in project 1. Project 1 will be available from Tuesday this week and we will discuss it during the lab sessions. 

In summary, this week we cover:

Material for the lecture on Monday September 1:

1. Linear Regression, ordinary least squares (OLS), Ridge and Lasso and mathematical analysis

2. Derivation of Gradient descent and discussion of implementations for

Material for the lab sessions on Tuesday and Wednesday (see at the end of these slides):

1. Technicalities concerning Ridge and Lasso linear regression.

2. Presentation and discussion of the first project. 

Reading suggestion:

Goodfellow et al, Deep Learning, introduction to gradient descent, see chapter 4.3 at https://www.deeplearningbook.org/contents/numerical.html

Rashcka et al, pages 37-44 and pages 278-283 with focus on linear regression.

Best wishes to you all,

Morten et al

p.s. For those of you who filled in the form searching for team mates, you will receive a suggestion on Monday, Sept 1.