Generalized Linear Models, e.g., Logistic Regression

 

Session 3: Generalized Linear Models (GLMs) — specifically Logistic Regression, which extends linear models to classification problems. This is the natural next step after regression with L2 loss and regularization.

1. Why Extend Linear Models?

• Linear regression works for continuous outcomes (like house prices).

• But many problems are classification:

  • Spam vs. not spam

  • Disease vs. healthy

  • Win vs. lose

We need a model that predicts probabilities between 0 and 1, not arbitrary continuous values.

2. Logistic Regression Model

Instead of predicting $y$ directly, we predict the probability:

$$P(y=1\mathrm{\mid }x)=\frac{1}{1+e^{-({\beta }_0+{\beta }_1{x}_1+\cdots +{\beta }_p{x}_p)}}$$

• This is the sigmoid function (S-shaped curve).

• It maps any linear combination of features into the range $[0,1]$.

• Interpretation: Probability that the outcome is “1” (positive class).

3. Loss Function (Cross-Entropy)

• Linear regression uses L2 loss (SSE).

• Logistic regression uses cross-entropy loss:

$$L(\beta )=-\sum _{i=1}^n[y_i\log ({\widehat{p}}_i)+(1-y_i)\log (1-{\widehat{p}}_i)]$$

• This penalizes wrong predictions more strongly when the model is confident but wrong.

• Ensures probabilities are well-calibrated.

4. Optimization

• No closed-form solution like OLS.

• We use iterative methods (e.g., gradient descent, Newton-Raphson).

• Regularization (Ridge, Lasso, Elastic Net) can also be applied to logistic regression.

5. Statistical Properties

• Interpretability: Coefficients represent log-odds changes.

  • Example: If ${\beta }_1=0.7$, then a one-unit increase in $x_1$ multiplies the odds of outcome by $e^{0.7}\approx 2$.

• Probabilistic output: Unlike regression, logistic regression gives probabilities, not raw values.

• Classification: By setting a threshold (e.g., 0.5), we classify into categories.

6. Example (Spam Detection)

Features:

• $x_1$ = number of links in email

• $x_2$ = presence of certain keywords

Model:

$$P(\text{spam}\mathrm{\mid }x)=\frac{1}{1+e^{-({\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2)}}$$

• If probability > 0.5 → classify as spam.

• If probability < 0.5 → classify as not spam.

7. Key Takeaways

• Logistic regression = linear model adapted for classification.

• Uses sigmoid function to map predictions into probabilities.

• Uses cross-entropy loss instead of L2 loss.

• Optimized iteratively, can include regularization.

• Coefficients are interpretable in terms of odds ratios.

✨ With this, you now see how linear models evolve from predicting continuous values (regression) to predicting probabilities (classification).

Alright, let’s continue into Session 4: Extensions of Generalized Linear Models (GLMs). This is where logistic regression expands into more complex situations like multiclass classification and count data modeling.

1. Multinomial Logistic Regression 🎨

• Logistic regression handles binary outcomes (yes/no, spam/not spam).

• Multinomial logistic regression extends this to multiple categories (e.g., predicting which sport someone likes: football, basketball, or tennis).

• Model idea:

$$P(y=k\mathrm{\mid }x)=\frac{e^{{\beta }_{0k}+{\beta }_{1k}{x}_1+\cdots +{\beta }_{pk}{x}_p}}{\sum _{j=1}^K{e}^{{\beta }_{0j}+{\beta }_{1j}{x}_1+\cdots +{\beta }_{pj}{x}_p}}$$

• Uses the softmax function to ensure probabilities across all classes sum to 1.

• Application: Text classification, image recognition, customer segmentation.

2. Poisson Regression 📊

• Used when the target variable is a count (number of events).

• Example: Number of hospital visits per year, number of clicks on a webpage.

• Model:

$$E[y\mathrm{\mid }x]=e^{{\beta }_0+{\beta }_1{x}_1+\cdots +{\beta }_p{x}_p}$$

• Ensures predictions are non-negative integers.

• Loss function: Based on the Poisson likelihood.

• Application: Epidemiology, traffic modeling, web analytics.

3. Other GLM Extensions

• Log-linear models: For categorical data analysis.

• Gamma regression: For modeling continuous, positive data (like insurance claims).

• Negative binomial regression: For count data with overdispersion (variance > mean).

4. Why GLMs Matter

• They unify regression and classification under one framework.

• Each GLM:

  • Uses a link function (sigmoid, log, softmax, etc.) to connect linear predictors to the outcome.

  • Adapts to different types of data (binary, categorical, counts, continuous positive).

• This flexibility makes GLMs a cornerstone of statistical modeling and machine learning.

5. Key Takeaways

• Logistic regression: Binary outcomes.

• Multinomial logistic regression: Multiple categories (softmax).

• Poisson regression: Count data.

• Other GLMs: Handle specialized data types.

• GLMs = powerful generalization of linear models for diverse real-world problems.

✨ With this, you now have a full picture:

• Session 1 → Linear regression with L2 loss.

• Session 2 → Regularization (Ridge, Lasso, Elastic Net).

• Session 3 → Logistic regression for classification.

• Session 4 → GLM extensions for multiclass and count data.

Would you like me to design a mini practice quiz (with multiple-choice questions) across Sessions 1–4 so you can test your understanding and reinforce what you’ve learned?

yes

Here’s a mini practice quiz covering Sessions 1–4 (Linear Models, Regularization, Logistic Regression, GLMs) so you can test your understanding and reinforce what you’ve learned.

📘 Practice Quiz: Linear Models & GLMs

Session 1: Linear Regression with L2 Loss

1. What does minimizing L2 loss (squared error) achieve in linear regression?

   • A) Maximizes variance of coefficients

   • B) Minimizes the sum of squared errors (SSE)

   • C) Ensures all coefficients are zero

   • D) Maximizes correlation between features

Session 2: Regularization

2. Which statement best describes Ridge regression?

   • A) It eliminates irrelevant features by setting coefficients to zero.

   • B) It shrinks coefficients smoothly but keeps all features.

   • C) It has no penalty term.

   • D) It always produces sparse models.

3. Lasso regression differs from Ridge regression because:

   • A) It uses an L2 penalty.

   • B) It requires fewer data points.

   • C) It can set some coefficients exactly to zero.

   • D) It has a closed-form solution.

Session 3: Logistic Regression

4. Logistic regression is used when:

   • A) The target variable is continuous.

   • B) The target variable is binary (yes/no).

   • C) The target variable is a count.

   • D) The target variable is categorical with more than two classes.

5. What loss function does logistic regression use?

   • A) L2 loss (SSE)

   • B) Cross-entropy loss

   • C) Hinge loss

   • D) Poisson likelihood

Session 4: GLM Extensions

6. Multinomial logistic regression uses which function to ensure probabilities sum to 1?

   • A) Sigmoid

   • B) Softmax

   • C) Log link

   • D) Identity link

7. Poisson regression is most appropriate when the target variable is:

   • A) Binary outcomes

   • B) Continuous values

   • C) Count data (number of events)

   • D) Ordinal categories

✅ Answer Key

1. B

2. B

3. C

4. B

5. B

6. B

7. C

✨ This quiz gives you a quick way to check your grasp of the concepts.

Perfect — let’s build a calculation-based practice quiz so you can strengthen the math side of Sessions 1–4. These will be short problems where you compute or interpret results, just like in real exercises.

🧮 Practice Quiz: Calculation & Interpretation

Session 1: Linear Regression

1. Suppose we have data:

   • $x=[1,2,3]$, $y=[2,4,5]$. Compute the slope $\widehat{{\beta }_1}$ using the formula:

$$\widehat{{\beta }_1}=\frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x}{)}^2}$$

(Hint: $\bar{x}=2$, $\bar{y}=3.67$).

Session 2: Regularization

2. In Ridge regression, the solution is:

$${\widehat{\beta }}_{ridge}=(X^{T}X+\lambda I{)}^{-1}{X}^{T}y$$

If $\lambda$ increases, what happens to the coefficients?

• A) They grow larger.

• B) They shrink toward zero.

• C) They remain unchanged.

• D) They oscillate randomly.

Session 3: Logistic Regression

3. A logistic regression model predicts:

$$P(y=1\mathrm{\mid }x)=\frac{1}{1+e^{-(-2+0.5x)}}$$

Compute the probability when $x=4$. (Hint: Plug in: $-2+0.5(4)=0$. Then sigmoid(0) = 0.5).

Session 4: GLM Extensions

4. In Poisson regression, the expected count is:

$$E[y\mathrm{\mid }x]=e^{{\beta }_0+{\beta }_{1}x}$$

If ${\beta }_0=1$, ${\beta }_1=0.2$, and $x=5$, compute $E[y\mathrm{\mid }x]$. (Hint: $1+0.2(5)=2$. Then $e^2\approx 7.39$).

✅ Answer Key

1. $\widehat{{\beta }_1}\approx 1.5$

2. B) They shrink toward zero.

3. $P(y=1\mathrm{\mid }x=4)=0.5$

4. $E[y\mathrm{\mid }x=5]\approx 7.39$

✨ These exercises give you hands-on practice with the formulas and interpretations.

Would you like me to create a larger exam-style set (10–15 questions mixing theory + calculations) so you can simulate a full test session for deeper practice?

yes

Perfect — let’s simulate a full exam-style set with a mix of theory and calculation questions across Sessions 1–4. This will feel like a real test, giving you both conceptual and numerical practice.

📘 Exam-Style Quiz: Linear Models & GLMs (15 Questions)

Session 1: Linear Regression with L2 Loss

1. Define the L2 loss function in linear regression and explain why squaring the errors is useful.

2. Given data points $(x,y)=\{(1,2),(2,3),(3,5)\}$, compute the slope $\widehat{{\beta }_1}$ and intercept $\widehat{{\beta }_0}$.

3. Explain the geometric interpretation of the OLS solution in terms of projection.

4. True/False: The OLS estimator is unbiased if the error terms have mean zero.

Session 2: Regularization (Ridge, Lasso, Elastic Net)

5. Write the Ridge regression loss function and explain the role of $\lambda$.

6. In Lasso regression, why can some coefficients become exactly zero?

7. Suppose Ridge regression is applied with a very large $\lambda$. What happens to the coefficients?

8. Compare Ridge and Lasso in terms of feature selection.

9. Elastic Net combines which two penalties, and why is this useful?

Session 3: Logistic Regression

10. Write the logistic regression model for $P(y=1\mathrm{\mid }x)$.

11. A logistic regression model is:

$$P(y=1\mathrm{\mid }x)=\frac{1}{1+e^{-(-1+0.8x)}}$$

Compute the probability when $x=2$.

12. What loss function does logistic regression minimize, and why is it preferred over L2 loss?

13. Interpret the coefficient ${\beta }_1=0.5$ in terms of odds ratio.

Session 4: GLM Extensions

14. In multinomial logistic regression, explain how the softmax function ensures probabilities sum to 1.

15. A Poisson regression model is:

$$E[y\mathrm{\mid }x]=e^{1+0.3x}$$

Compute the expected count when $x=4$.

✅ Answer Key (Step-by-Step)

1. $L(\beta )=\sum (y_i-{\widehat{y}}_i{)}^2$. Squaring penalizes large errors more and ensures differentiability.

2. $\bar{x}=2$, $\bar{y}=3.33$. Slope $\widehat{{\beta }_1}=\frac{(1-2)(2-3.33)+(2-2)(3-3.33)+(3-2)(5-3.33)}{(1-2{)}^2+(2-2{)}^2+(3-2{)}^2}=\frac{1.33+0+1.67}{1+0+1}=1.5$. Intercept $\widehat{{\beta }_0}=3.33-1.5(2)=0.33$.

3. OLS = projection of $\mathbf{y}$ onto the column space of $\mathbf{X}$.

4. True.

5. $L(\beta )=\sum (y_i-{\widehat{y}}_i{)}^2+\lambda \sum {\beta }_j^2$. $\lambda$ controls shrinkage.

6. Because the L1 penalty creates sharp corners in the constraint region, allowing exact zeros.

7. Coefficients shrink toward zero.

8. Ridge keeps all features; Lasso selects by eliminating some.

9. Elastic Net = L1 + L2 penalties. Useful for correlated + sparse features.

10. $P(y=1\mathrm{\mid }x)=\frac{1}{1+e^{-({\beta }_0+{\beta }_{1}x)}}$.

11. Plug in: $-1+0.8(2)=0.6$. Sigmoid(0.6) ≈ 0.645.

12. Cross-entropy loss. Preferred because it models probabilities correctly and penalizes confident wrong predictions.

13. Odds ratio = $e^{0.5}\approx 1.65$. A one-unit increase in $x$ multiplies odds by 1.65.

14. Softmax normalizes exponentials: each probability = exponential of score / sum of exponentials.

15. $E[y\mathrm{\mid }x=4]=e^{1+0.3(4)}=e^{2.2}\approx 9.03$.

✨ This exam-style set gives you a balanced mix of theory, interpretation, and calculation.