Paper: Regression Analysis III

(1)

Subject: Statistics

Paper: Regression Analysis III

Module: Logit and probit models

(2)

Development Team

Principal investigator: Dr. Bhaswati Ganguli,Professor, Department of Statistics, University of Calcutta

Paper co-ordinator: Dr. Bhaswati Ganguli,Professor, Department of Statistics, University of Calcutta

Content writer: Sayantee Jana, Graduate student, Department of Mathematics and Statistics, McMaster University Sujit Kumar Ray,Analytics professional, Kolkata

Content reviewer: Department of Statistics, University of Calcutta

Regression Analysis III 2 / 14

(3)

Introduction

I Q : When do we use Logistic Regression ?

I A : When the response variable is binary and the explanatory variable(s) is/are numeric and/or categorical. A logistic regression model can have one or more than one predictor variable.

I Logistic regression is often used because the relationship between the discrete response variable and a predictor is non-linear.

I Example¹ : The probability of heart disease changes very little with a 10 point difference among people with low blood pressure, but a 10 point change can mean a drastic change in the probability of heart disease in people with high blood pressure.

1Agresti, A. (2014). Categorical data analysis. John Wiley Sons.

(4)

Introduction

(5)

Introduction

(6)

Introduction

(7)

Assumptions of logistic regression model

I The only “real” limitation of logistic regression is that the outcome must be discrete..

I If the distributional assumptions are met than discriminant function analysis may be more powerful, although it has been shown to overestimate the association using discrete

predictors.

I If the outcome is continuous then multiple regression is more powerful given that the assumptions are met.

I Ratio of cases to variables - using discrete variables requires that there are enough responses in every given category.

I If there are too many cells with no responses parameter estimates and standard errors will be likely to blow up.

(8)

predictors.

(9)

predictors.

(10)

predictors.

(11)

predictors.

(12)

Assumptions Continued ...

I Too many cells can make groups perfectly separable (e.g.

multicollinear) which will make maximum likelihood estimation impossible.

I Linearity in the logit - the regression equation should have a linear relationship with the logit form of the discrete variable.

There is no assumption about the predictors being linearly related to each other.

I Absence of multicollinearity.

I No outliers.

I Independence of errors - assumes a between subjects design.

There are other forms if the design is within subjects.

(13)

I No outliers.

(14)

I No outliers.

(15)

I No outliers.

(16)

I No outliers.

(17)

I No outliers.

(18)

The logistic regression model

I Y : binary response variable

I X = (x1, x2, ..., xp): explanatory variables

I Linear Probability Model : Y_i=X_i⁰β+u_i

Yi =

(1 if an event happens

0 if an event does not happen

I Pi=P[Yi = 1|X_i],0≤Pi≤1

I E(Yi|X_i) =1.Pi+ 0.(1−Pi)=Pi

I P_i=X_i⁰β ⇒ does not guarantee 0≤P_i ≤1 and hence violates the law of probability.

(19)

Yi =

(20)

Yi =

(21)

Yi =

(22)

Yi =

(23)

Utility function

I We define, Ui =X_i⁰β

I latent variable : U^∗ ∼F(.)

I F(.) is likely to be a symmetric bell-shaped distribution.

Yi=

(1 if Ui ≥U^∗ 0 if Ui < U^∗

P_i = P[Y_i= 1|X_i]

= P[Ui≥U^∗ |Xi]

= P[U^∗ ≤X_i⁰β |Xi]

= F(X_i⁰β)

(24)

Logit model

I F ∼Φi.e. Pi = Φ(X_i⁰β) → Probit model

I F ∼logistic distribution i.e. Pi =F(X_i⁰β) = ¹

(1+e^−X

0 iβ)

I log(_1−P^Pⁱ

i) =X_i⁰β : logit model

I Merits of the logit model :

I Modeling the log of odds of an event.

I F(.)is symmetric.

(25)

Logit model

(1+e^−X

0 iβ)

I log(_1−P^Pⁱ

I F(.)is symmetric.

(26)

Logit model

(1+e^−X

0 iβ)

I log(_1−P^Pⁱ

I F(.)is symmetric.

(27)

Logit model

(1+e^−X

0 iβ)

I log(_1−P^Pⁱ

I F(.)is symmetric.

(28)

Calculating Odds Ratio

Calculating Odds Ratio : From a contingency table.

Let us consider a 2×2 table of counts.

Table I : Response variableY vs. Predictor variable X Predictor Response variable

Variable Diseased Non-diseased

Exposed a b

Unexposed c d

OR= a×d b×c

(29)

Calculating Odds Ratio

Calculating Odds Ratio : From a contingency table.

Let us consider a 2×2 table of counts.

Table I : Response variableY vs. Predictor variable X Predictor Response variable

Variable Diseased Non-diseased

Exposed a b

Unexposed c d

OR= a×d b×c

(30)

Log of odds

I Let π₀ =P(E|D),π₁ =P(E|N D)

I log(_1−π^π ) = intercept + disease status ×β

I disease status= 1, if diseased disease status=0, if non diseased

I log of odds(π0)= intercept +β log of odds(π1)= intercept

(31)

Log of odds

(32)

Log of odds

(33)

Log of odds

(34)

Calculating OR : from a logit model

I log of odds ratio comparing exposed to unexposed

= log of odds for exposed - log odds for unexposed

=(intercept + β×1)-(intercept +β×0)=β

I H₀:β= 0 ≈log(OR) = 0 ≈OR= 1

I Thus when we have multiple covariates and multiple ORs to calculate and test for association between each of the

predictor and the response variable, then instead of calculating each of the ORs from the corresponding contingency table it is more convenient to just test each parameter in a logit model.

I This convenience also applies when we have multiple categories for any of the predictor variables.

(35)

I H₀:β= 0 ≈log(OR) = 0 ≈OR= 1

(36)

I H₀:β= 0 ≈log(OR) = 0 ≈OR= 1

(37)

I H₀:β= 0 ≈log(OR) = 0 ≈OR= 1

(38)

Summary

I In a logistic model we require the response variable to be a binary variable.

I The predictor variables can be discrete or continuous.

I In logistic regression we model the log of odds of an event.

I Hence a logistic regression model is easy to interpret.

I Testing the parameter of a logit model is equivalent to testing the odds ratio.

I It is more convenient to test the parameters of logit model than calculating many Odds Ratios from contingency tables and testing for them, when we have multiple predictor variables or multiple categories of the predictor variable.

(39)

Summary

(40)

Summary

(41)

Summary

(42)

Summary

(43)

Summary

(44)

Example 1²

mydata=read.csv("http://www.ats.ucla.edu/stat/data/

binary.csv")

## view the first few rows of the data head(mydata)

# logistic regression

mylogit=glm(admit ~ as.factor(rank),data=mydata, family="binomial")

summary(mylogit) mylogit$coefficients

ct=xtabs(~admit +rank, data = mydata)

# install.packages(epitools)

require(epitools) # contains the function oddsratio oddsratio((ct))$measure

exp(mylogit$coefficients)

2Source of data: http://www.ats.ucla.edu/stat/r/dae/logit.htm

Regression Analysis III Sample R script to complement this module 13 / 14

(45)

Example 2³

# install.packages("coin")

require(coin) ## contains the dataset alzheimer data(alzheimer)

head(alzheimer)

## graphical display

layout(matrix(1:2, ncol = 2))

spineplot(disease ~ smoking, data = alzheimer, subset = gender == "Male",main = "Male") spineplot(disease ~ smoking, data = alzheimer,

subset = gender == "Female",main = "Female")

mod2=(glm(as.factor(disease)~as.factor(gender)+as.factor(smoking),data=alzheimer,family=binomial(link=logit))) summary(mod2);mod2$coefficients

ct=xtabs(~disease+gender+smoking, data = alzheimer);ct

3help file for the dataset alzheimer in library coin, R documentation