Likelihood Equation of MLE

Saurav De

Department of Statistics Presidency University

Likelihood Equation of MLE

MLE and Invariance Property

Let ˆθbe MLE of θ.Then for the parametric function g(θ) : Ω→Γ; MLE is g(ˆθ).

Proof. Let us define Ω_γ ={θ:g(θ) =γ}.This means Ω = [

γ∈Γ

Ω_γ. Again let Mx(γ) = sup

θ∈Ω_γ

Lx(θ) = Likelihood function induced byg. We are to find ˆγ at which Mx(γ) is maximised.

Now Mx(ˆγ0) = sup

θ∈Ω_γˆ

0

Lx(θ)≥Lx(ˆθ)

θ∈Ω_γˆ

0

where Ω_γˆ0 ={θ:g(θ) = ˆγ₀}.As g(ˆθ) = ˆγ₀ so ˆθ ∈ Ω_γˆ0 Again

M_x(ˆγ₀)≤sup

ˆγ∈Γ

M_x(ˆγ) = sup

ˆγ∈Ω

sup

θ∈Ω_γˆ

0

L_x(θ)

= sup

θ∈Ω

Lx(θ) =Lx(ˆθ)

Likelihood Equation of MLE

Therefore

M_x(ˆγ₀) =L_x(ˆθ) = sup

γ∈Γ

M_x(γ).

Hence ˆγ₀ is the MLE ofγ, i.e. g(ˆθ)(= ˆγ₀) is the MLE of g(θ)(=γ).

Proved

Ex.3Let X1,X2, . . . ,Xn∼Bin (1,p) ; 0≤p ≤1 Then Vp(X) =p(1−p)(=g(p)) and pˆ_MLE =Xn. By invariance property MLE ofV_p(X) is

g( ˆpMLE) = ˆpMLE(1−pˆMLE) =Xn(1−Xn).

Application 4. Suppose that n observations are taken on a random variable X with distribution N(µ,1),but instead of recording all the observations, one notes only whether or not the observation is less than 0.

If{X <0}occursm(<n) times, find the MLE ofµ.

Let X₁,X₂, . . . ,X_n∼N(µ,1).

Let θ=P_µ[X₁ <0] = Φ(−µ) = 1−Φ(µ).

This means µ=−Φ⁻¹(θ), a continuous function ofθ.

Likelihood Equation of MLE

Y_i = 1 ifX_i <0

= 0 ifXi ≥0 Then Y1,Y2, . . . ,Yn∼Bin (1, θ) ; 0≤θ≤1, Now the MLE ofθ is Y = ¹_n

n

X

i=1

Y_i = ^#{X_n^i<0} = ^m_n.(See the Application 3).

Hence by the invariance property the MLE of µis −Φ⁻¹(^m_n).

Application. A commuter trip consists of first riding a subway to the bus stop and then taking a bus. The bus she should like to catch arrives uniformly over the intervalθ1 toθ2.She would like to estimate bothθ1 and θ2 so that she would have some idea about the time she should be at the bus stop (θ₁) and she should have too late and wait for the next bus (θ₂).

Over an 8-day period she makes certain to be at the bus stop so early not to miss the bus and records the following arrival time of the bus.

5:15 PM , 5:21 PM , 5:14 PM , 5:23 PM , 5:29 PM , 5:17 PM , 5:15 PM , 5:18 PM

Estimate θ1 andθ2.Also give the MLEs for the mean and the variability of the arrival distribution.

Likelihood Equation of MLE

Solution: LetT : arrival time of the bus. Then according to the question T ∼R(θ1, θ2).

Let T₁,T₂, . . . ,T_n be n independent random arrival times of the bus.

Then the likelihood of θ= (θ₁, θ₂) is

L(θ) = 1

(θ₂−θ₁)ⁿ ifθ₁ ≤t_i ≤θ₂;i = 1,2, . . . ,n

= 0 otherwise i.e. L(θ) = 1

(θ₂−θ₁)ⁿ ifθ₁≤t₍₁₎<t_(n)≤θ₂

= 0 otherwise

where t₍₁₎= min{t₁,t₂, . . . ,t_n}andt_(n)= max{t₁,t₂, . . . ,t_n}.

Under θ1≤t₍₁₎ <t_(n)≤θ2 , t_(n)−t₍₁₎ ≤θ2−θ1. Hence L(θ1, θ2) = _(θ ¹

2−θ1)ⁿ ≤ _(t ¹

(n)−t₍₁₎)ⁿ =L(t₍₁₎,t_(n)).

Thus MLE of (θ₁, θ₂) is (T₍₁₎,T_(n)), whereT₍₁₎ andT_(n) are the minimum and maximum order statistic respectively.

Now E(T) = ^θ1+θ₂ ² andV(T) = ^(θ2−θ₁₂¹⁾² are two continuous functions of (θ₁, θ₂).

So MLE of Mean : ^T(1)+T₂ ⁽ⁿ⁾ and of variability : ^{(T(n)√−T(1))}

12 (from invariance property of MLE)

Likelihood Equation of MLE

Computation using R :

(The minute component of the time has been considered as the data) R Code and Output :

> s a m p = c (15 ,21 ,14 ,23 ,29 ,17 ,15 ,18)# g i v e n s a m p l e

> n = l e n g t h ( s a m p )# s i z e of the s a m p l e

> n [1] 8

> MLE _ t h e t a 1 = min ( s a m p )# MLE of the p a r a m e t e r s

> MLE _ t h e t a 2 = max ( s a m p )

> MLE _ t h e t a 1 [1] 14

> MLE _ t h e t a 2 [1] 29

> MLE _ M e a n =( MLE _ t h e t a 1 + MLE _ t h e t a 2 ) / 2# MLE of m e a n

> cat ( " The m e a n a r r i v a l t i m e is : " , MLE _ Mean , " m i n u t e s a f t e r 5 pm .\ n " ) The m e a n a r r i v a l t i m e is : 2 1 . 5 m i n u t e s a f t e r 5 pm .

> MLE _ Var =( MLE _ theta2 - MLE _ t h e t a 1 ) / s q r t ( 1 2 )# MLE of v a r i a b i l i t y

> MLE _ Var [1] 4 . 3 3 0 1 2 7

Likelihood Equations and Related Discussions lx(θ) = lnLx(θ) is called log-likelihood function ofθ.

Likelihood Equation: ^∂l_∂θ^x(θ) = 0.

Any MLE is a root of the likelihood equation.

Any root may be local minimaor local maxima.

Possible verification for the root ˆθto be an MLE: ^∂2_∂θ^lx(θ)2 |_θ=ˆθ <0.

Ifθ = (θ1, . . . , θs)⁰,the likelihood equations: ^∂lx(θ)

∂θi = 0 i = 1, . . . ,s.

Possible verification for the root ˆθto be an MLE of θ: The Hessian matrix ∂²lx(θ)

∂θ_i∂θ_j |_θ

i=ˆθ_i,θ_j=ˆθ_j

is negative definite.

Likelihood Equation of MLE

Result: Let T be a sufficient statistic for the family of distributions {f_θ:θ∈Ω}.If a unique MLE of θexists, it is a (nonconstant) function of T.If a MLE ofθ exists but is not unique, one can find a MLE that is a function of T.

Proof. Since T is sufficient, from Neyman-Fisher factorisability we can write

L(θ) =fθ(x) =gθ(T(x))h(x)

for all x, all θand some h andgθ, where L(θ) is the likelihood function of θ andf_θ(x) is the joint pmf or pdf of sample observations x= (x1, . . . ,xn).

IfL(θ) is maximised by a unique MLE ˆθ, naturally it will also maximise the functiong_θ(T(x)).

=⇒ θˆshould be a function of T.

If MLE ofθ exists but is not unique, then∃ some MLEθ that can be expressed as a function ofT.

Proved

Likelihood Equation of MLE

Result: Under regular estimation case (i.e. the situation where all the regularity conditions of Cramer-Rao Inequality hold) if an estimator ˆθ ofθ attains the Cramer-Rao Lower Bound CRLBfor the variance, the

likelihood equation has a unique solution ˆθ that maximises the likelihood function.

Proof. Let L(θ|x) denote the likelihood function of real-valued parameter θ given the sample observationsx= (x1, . . . ,xn).Iffθ(x) denotes the joint pmf or pdf of x, from the equality condition in CR inequality we get

∂

∂θlogfθ(x) =k(θ)(ˆθ(x)−θ)

that is

∂

∂θlogL(θ|x) =k(θ)(ˆθ(x)−θ) (∗) with probability 1.

=⇒ the likelihood equation _∂θ^∂ logL(θ|x) = 0 has the unique solution θ= ˆθ.

Differentiating both sides of (∗) again with respect to θwe get

∂²

∂θ² logL(θ|x) =k⁰(θ)(ˆθ−θ)−k(θ).

Hence

∂²

∂θ²logL(θ|x)|_θ=ˆθ=−k(θ) (∗∗)

Likelihood Equation of MLE

Now, If T is an unbiased estimator of a real-valued estimable parametric functiong(θ) which is differentiable at least once, from CR regularity conditions we directly get

Z

T(x))fθ(x)dx=g(θ) Or, differentiating both sides with respect to θ we get

Z

T(x))∂

∂θlogf_θ(x)dx=g⁰(θ) In particular choosing g(θ) =θand noting that E_θ∂

∂θlogL(θ|X)

= 0 we get

E_θ

(T(X)−θ) ∂

∂θlogL(θ|X)

= 1.

Finally substituting

T(X)−θ= [k(θ)]⁻¹ ∂

∂θ logL(θ|X)

(looking at (∗) as T(X)is just like θ(X) by its definition) we getˆ [k(θ)]⁻¹Eθ

∂

∂θlogL(θ|X) 2

= 1 That is

k(θ) =E_θ ∂

∂θ logL(θ|X) 2

>0 by regularity condition Thus from (∗∗) the S.O.C. for maximising L(θ) holds. Hence proved.

Likelihood Equation of MLE

Result: Suppose ^∂2_∂θ^lx(θ)2 ≤0 ∀ θ∈Ω.Then ˆθ satisfying ^∂l_∂θ^x(θ) = 0 is the global maxima.

Proof. l_x(θ) =l_x(ˆθ) + (θ−θ)ˆ ^∂l_∂θ^x(θ)|_θ=ˆθ+ ^(θ−₂^θ)ˆ2∂2_∂θ^lx(θ)2 |_θ=θ^∗ θ^∗∈(ˆθ, θ).

Note that the RHS ≤l_x(ˆθ) because, in RHS the 2nd factor of 2nd term vanishes and the 2nd factor of the third term ≤0.Hence proved.

Result: Suppose

(i) ^∂l_∂θ^x(θ) = 0 iff θ= ˆθ.

(ii) ^∂2_∂θ^lx(θ)2 |_θ=ˆθ <0.And

(iii) θˆis an interior point of an intervalI ⊂Ω.

Then ˆθ is the global maxima.

Proof. If possible suppose ˆθ^∗ is such thatlx(ˆθ^∗)>lx(ˆθ).Then there must be a local minima in between the local maxima ˆθ and ˆθ^∗.This means for that minima point also ^∂l_∂θ^x(θ) = 0,which is a contradiction to the

supposition (i). Hence proved.

Likelihood Equation of MLE

Ex.1Let X1,X2, . . . ,Xn∼N(µ, σ²) independently.

Then l_x(θ) = constant - X

(x_i−µ)²

2σ² −ⁿ₂lnσ².

So ^∂l_∂µ^x(θ) = 0 and ^∂l_∂σ^x(θ)2 = 0 imply ˆµ=x , σˆ²= _n¹X

(x_i −x)²=s². Also check that ^∂_∂µ∂σ^2lx(θ)2|_(µ,σ2)=(x,s²)= 0,

∂²lx(θ)

∂µ² |_(µ,σ2)=(x,s²)=−_sⁿ₂ , _∂∂(σ^∂2lx(θ)2)²|_(µ,σ2)=(x,s²)=−_sⁿ₄. So the Hessian matrix

H =

−n/s² 0 0 −n/2s⁴

is negative definite. Hence (X,S²) is the global maxima point and is the MLE of (µ, σ²).

Likelihood Equation of MLE

Aliter: Ifψ(x) =x−1−lnx then ψ⁰(x) = 1−1/x , ψ⁰⁰(x) = 1/x²>0.

Therefore ψ(x) is minimum atx = 1 and minψ(x) = 1−1−0 = 0.Based on this result we can write

lx(ˆµ,σˆ²)−lx(µ, σ²) = X

(xi−µ)²

2σ² +ⁿ₂lnσ²−ⁿ₂ −ⁿ₂lns²

≥ _2σ^ns2₂ − ⁿ₂ln_σ^s22 − ⁿ₂ = ⁿ₂h

s²

σ² −1−ln^s_σ²2

i≥0.

TRY YOURSELF!

M4.1. LetX₁,X₂, . . . ,X_n∼Lognormal(µ, σ²) independently. Then MLE of (µ, σ²) is (Y,S^∗2) whereY = logX and (Y,S^∗2) = (sample mean , sample variance(with divisor n)) onY.

Hint. IfX ∼Lognormal(µ, σ²) thenY = logX ∼N(µ, σ²).Now proceed as in Ex. 1.

M4.2. (continuation)If inM4.1. µ= 0, find the MLE of σ²

M4.3. Consider a random sample of sizen from Exponential (mean = β).

It is given only thatk, 0<k <n, of thesen observations are≤M, where M is a known positive number. Find the MLE of β.

Likelihood Equation of MLE

TUTORIAL DISCUSSION :

Overview to the problems from MODULE 4. . .

M4.2. IfX ∼Lognormal(0, σ²) then Y = logX ∼N(0, σ²).

Given y= (y₁, . . . ,y_n), the loglikelihood function ofσ² is l(σ²) = constant −n

2σ²− 1 2σ²

n

X

i=1

y_i²

Now using maxima-minima principle from F.O.C. we get σ² = ¹_n

n

X

i=1

y_i²= ˆσ² (say) is the only solution of the likelihood equation.

Also _∂(σ^∂22)²l(σ²)|_σ2=ˆσ² =−_2ˆⁿ_σ2 <0.

Moreover ˆσ² is an interior point of the parameterspace Ω = (0,∞).

=⇒ σˆ² is the point of global maxima of the likelihood function i.e. the unique MLE of σ².

Likelihood Equation of MLE

M4.3. LetX₁,X₂, . . . ,X_n∼Exponential (mean =β).

DefineY_i = 1(0) ifX_i ≤M (X_i >M).

Then Y1,Y2, . . . ,Yn∼Bin (1, θ) ; 0< θ <1, where θ=P_β[X1 ≤M] = 1−exp

h

−^M_βi

.This meansβ =−_log(1−θ)^M , a continuous function of θ.

Now the MLE ofθ is Y = ¹_n

n

X

i=1

Y_i = ^#{Xi_n^≤M} = ^k_n.(See the Application 4).

Hence by the invariance property the MLE of β is− ^M

log(1−^k

n).