Linear Regression - In Depth (Notes)

• What is regression?
• What are the sample mean, sample variance, MSE, and RSS formulas? How are they similar? Why are they relevant for regression?
• What is the residual term?
• How can we explain these residuals? (Ans: Statistical Fluke, causal, and confounding variable, but explain)
• Explain Multiple Linear Regression in your own words.
• Explain Linear Basis Function Regression in your own words.
• Express the Linear Basis Function Model in both vector form with Y_hat_i as the target variable, and in matrix form with y_hat as the target variable!
• Why do we represent our Regression in a design matrix and weight vector form?
• What is the “recipe” for linear regression?
• Our Linear Regression sheet states that the loss function for simple linear regression is “convex”. Elaborate on that, and how do we use this to our advantage

• Given this MSE find the partial derivative of w_0 and w_1 and set it to 0, write down the expression. Explain what the output implies

$$ \frac{\operatorname{MSE}\left(w_0^, w_1^\right)}{\sigma_y^2}=1-\frac{\sigma_{x y}^2}{\sigma_x^2 \sigma_y^2} $$

• How did we arrive to this expression above?

• What does the right-hand expression represent? What’s the left-hand? What’s the next step and why?
• What is R2? What is it for?
• What would you expect the R2 value to look like if we took a simple regression on sample data that follows a horizontal line with minimal variance?
• What is OLS and how does it differ from finding the partial derivative for simple regression MSE