Module: Principal Components Analysis -

(1)

Subject: Statistics

Paper: Multivariate Analysis

Module: Principal Components Analysis -

(2)

Development Team

Principal investigator: Dr. Bhaswati Ganguli, Professor, Department of Statistics, University of Calcutta

Paper co-ordinator: Dr. Sugata SenRoy,Professor, Department of Statistics, University of Calcutta

Content writer: Souvik Bandyopadhyay, Senior Lecturer, Indian Institute of Public Health, Hyderabad

Content reviewer: Dr. Kalyan Das,Professor, Department of Statistics, University of Calcutta

(3)

Development Team

(4)

Development Team

(5)

Development Team

(6)

Principal Components from Standardized Variables

I Let X= [X1, X2, . . . , Xm]⁰ have mean µ= [µ1, µ2, . . . , µm]⁰ and dispersion matrixΣ = ((σ_jj))with eigenvalues

λ₁≥λ₂ ≥ · · · ≥λ_m≥0.

Define the standardized variables Z_j = (Xj−µj)

√σ_jj , j= 1, . . . , m.

I Fot Z= [Z₁, Z₂, . . . , Z_m]⁰ andV, a diagonal matrix with elements σ_jj,

Z=V^−1/2(X−µ)

I Clearly E(Z) = 0and

Cov(Z) =V^−1/2ΣV^−1/2 =ρ.

(7)

λ₁≥λ₂ ≥ · · · ≥λ_m≥0.

√σ_jj , j= 1, . . . , m.

Z=V^−1/2(X−µ)

Cov(Z) =V^−1/2ΣV^−1/2 =ρ.

(8)

λ₁≥λ₂ ≥ · · · ≥λ_m≥0.

√σ_jj , j= 1, . . . , m.

Z=V^−1/2(X−µ)

Cov(Z) =V^−1/2ΣV^−1/2 =ρ.

(9)

Standardized Variables

I The principal components ofZmay be obtained from the eigenvectors of the correlation matrix ρof X.

I All the previous results will be valid since the variance of each Z_i is unity.

I Y_j will be used to denote thejth principal component and (λj,e_j) for the eigenvalue-eigenvector pair from eitherρ orΣ.

I However, the (λ_j,e_j)derived from Σare in general not the same as the ones derived from ρ.

(10)

(11)

(12)

(13)

I Result 4: The jth principal component of the standardized variables Z⁰ = [Z1, Z2, . . . , Zm]with Cov(Z) =ρ, is given by,

Yj =ej0

Z=ej0

V^−1/2(X−µ), j= 1,2, . . . , m Moreover,

m

X

j=1

Var(Y_j) =

m

X

j=1

Var(Z_j) =m.

and

ρYj,Zk =ejkλj j, k= 1,2, . . . , m.

In this case (λ1,e1),(λ2,e2), . . . ,(λm,em) are the eigenvalue-eigenvector pairs of ρ, with

(14)

I The proof of Result 4 follows from Results 1, 2 and 3 with Z₁, Z₂, . . . , Z_m in place of X₁, X₂, . . . , X_m andρin place of Σ.

I We know that the total population variance (for the

standardized variables) is simply m, the sum of the diagonal elements of the matrix ρ.

I Therefore, Proportion of standardized population variance due to the jth principal component

=λ_j/m, j = 1,2, . . . , m, whereλj’s are the eigenvalues ofρ.

(15)

(16)

(17)

Geometrical Interpretation

I SupposeX is distributed as Nm(µ,Σ). Then the density of X is constant on the µcentered ellipsoids,

(x−µ)⁰Σ⁻¹(x−µ) =c² . . .(∗)

I This have axes ±cp

λjej, j = 1,2, . . . , m, where the (λ_j,e_j) are the eigenvalue-eigenvector pairs ofΣ.

I A point lying on the jth axis of the ellipsoid will have coordinates proportional toej = [ej1, ej2, . . . , ejm]in the coordinate system that has originµand axes that are parallel to the original axesx₁, x₂, . . . , x_m.

(18)

(x−µ)⁰Σ⁻¹(x−µ) =c² . . .(∗)

(19)

(x−µ)⁰Σ⁻¹(x−µ) =c² . . .(∗)

(20)

I Settingµ= 0in (∗) we can write, c² =x⁰Σ⁻¹x= 1

λ1

(e10

x)²+ 1 λ2

(e20

x)²+· · ·+ 1 λm

(em0

x)² wheree10x,e20x, . . . ,em0x are recognized as the principal components ofx.

I Settingy₁ =e₁⁰x, y₂=e₂⁰x, . . . , y_m =e_m⁰x, we have c²= 1

λ1

y²₁+ 1 λ2

y²₂+· · ·+ 1 λm

y²_m

I This equation defines an ellipsoid (since λ1, λ2, . . . , λm are positive) in a coordinate system with axes y₁, y₂, . . . , y_m lying

(21)

λ1

(e10

x)²+ 1 λ2

(e20

x)²+· · ·+ 1 λm

(em0

λ1

y²₁+ 1 λ2

y²₂+· · ·+ 1 λm

y²_m

(22)

λ1

(e10

x)²+ 1 λ2

(e20

x)²+· · ·+ 1 λm

(em0

λ1

y²₁+ 1 λ2

y²₂+· · ·+ 1 λm

y²_m

(23)

I Ifλ1 is the largest eigenvalue, then the major axis lies in the direction e₁.

I The remaining minor axes lies in the directions defined by e2, . . . ,em.

I The principal components

y1 =e10x, y2=e20x, . . . , ym=em0xlie in the directions of the axes of a constant density ellipsoid.

I Therefore any point on thejth ellipsoid axis has xcoordinate proportional toe_j⁰= [e_j1, e_j2, . . . , e_jm]and necessarily

principal component coordinates of the form [0, . . . ,0, yj,0, . . . ,0].

I Whenµ6= 0, it is the mean-centered principal component

(24)

(25)

(26)

(27)

(28)

Special Covariance Structures

I Case I : Suppose Σbe the diagonal matrix

Σ =







σ11 0 . . . 0

0 σ22 . . . 0 ... ... . .. ...

0 0 . . . σmm







. . .(i)

I Settinge_j = [0, . . . ,0,1,0, . . . ,0], with 1 in the jth position, we observe that







σ₁₁ 0 . . . 0

0 σ₂₂ . . . 0

... ... . .. ...

0 0 . . . σ_mm











 0

... 0 1 0 ..





=





 0

... 0 σjj

0 ..





or Σej =σjjej

(29)

I Case I : Suppose Σbe the diagonal matrix

Σ =







σ11 0 . . . 0

0 σ22 . . . 0 ... ... . .. ...

0 0 . . . σmm







. . .(i)

I Settinge_j = [0, . . . ,0,1,0, . . . ,0], with 1 in the jth position, we observe that







σ₁₁ 0 . . . 0

0 σ₂₂ . . . 0 ... ... . .. ...

0 0 . . . σ_mm











 0

... 0 1 0 ..





=





 0

... 0 σjj

0 ..





or Σej =σjjej

(30)

I (σ_jj,e_j) is the jth eigenvalue-eigenvector pair.

I Since the linear combination e_j⁰X=X_j, the set of principal components is just the original set of uncorrelated random variables.

I For a covariance matrix like(i) nothing is gained by extracting the principal components.

I IfX∼Nm(µ,Σ), the contours of constant density are

ellipsoids whose axes already lie in the directions of maximum variation.

I Consequently there is no need to rotate the coordinate system.

(31)

(32)

(33)

(34)

(35)

I Standardization does not substantially alter the situation for theΣ in(i). In that caseρ=I, them×m identity matrix.

I Clearly ρej = 1ej, so that the eigenvalue 1 has multiplicitym ande_j⁰= [0, . . . ,0,1,0, . . . ,0], j= 1,2, . . . , m, are

convenient choices for the eigenvectors.

I The principal components determined from ρare also the original variables Z₁, Z₂, . . . , Z_m.

I Since the eigenvalues are equal, the multivariate normal ellipsoids of constant density are spheroids.

(36)

(37)

(38)

(39)

I Case II : Consider another covariance matrix, that often describes the correspondence among certain biological variables (e.g. size of living organisms)

Σ =







σ² ρσ² . . . ρσ² ρσ² σ² . . . ρσ² ... ... . .. ... ρσ² ρσ² . . . σ²







. . .(ii)

The resulting correlation matrix

ρ=







1 ρ . . . ρ ρ 1 . . . ρ ... ... . .. ...







. . .(iii)

(40)

I The matrix in (iii) implies that the variablesX₁, X₂, . . . , X_m are equally correlated.

I The meigenvalues of the correlation matrix can be divided into two groups. When ρ is positive the largest is

λ₁ = 1 + (m−1)ρ . . .(iv)

with associated eigenvector,

e10 = 1

√m, 1

√m,· · ·, 1

√m

. . .(v) The remaining m−1 eigenvalues are

λ₁=λ₂ =· · ·=λ_p = 1−ρ

(41)

λ₁ = 1 + (m−1)ρ . . .(iv)

e10 = 1

√m, 1

√m,· · ·, 1

√m

λ₁=λ₂ =· · ·=λ_p = 1−ρ

(42)

λ₁ = 1 + (m−1)ρ . . .(iv)

e10 = 1

√m, 1

√m,· · ·, 1

√m

λ₁=λ₂ =· · ·=λ_p = 1−ρ

(43)

I One choice for these eigenvectors is e20 =

h√1

1×2,^√⁻¹_1×2,0,· · ·,0 i

e30 = h√1

2×3,^√_2×3¹ ,^√⁻²_2×3,0,· · ·,0 i

... ...

ej0 =

√ 1

(j−1)j,· · · ,√ ¹

(j−1)j,√^−(j−1)

(j−1)j,0,· · · ,0

... ...

e_m⁰ =

√ 1

(m−1)m,· · · ,√ ¹

(m−1)m,√^−(m−1)

(m−1)m

(44)

I The first principal component,

Y₁=e₁⁰Z= 1

√m

m

X

j=1

Z_j

is proportional to the sum of the m standardized variables.

I Might be regarded as an index with equal weights.

I This principal component explains a proportion λ1

m = 1 + (m−1)ρ

m =ρ+ 1−ρ

m . . .(vi)

of the total population variation.

(45)

Y₁=e₁⁰Z= 1

√m

m

X

j=1

Z_j

m = 1 + (m−1)ρ

m =ρ+ 1−ρ

m . . .(vi)

(46)

Y₁=e₁⁰Z= 1

√m

m

X

j=1

Z_j

m = 1 + (m−1)ρ

m =ρ+ 1−ρ

m . . .(vi)

(47)

I λ1/m≈ρ for ρ close to 1 orm .

For example, ifρ= 0.80 andm= 5, the first component explains 84%of the total variability.

I Whenρ is near 1, a very small proportion of total variance is explained by the lastm−1components.

I In this case we retain only the first principal componentY1.

(48)

(49)

(50)

Scree Plot

I A Scree Plot is an useful visual aid to determine the appropriate number of principal components.

I A scree plot is a plot of λˆ_j versus j where λˆ₁≥ˆλ₂ ≥ · · · ≥λˆ_j ≥0.

I This would be a downward sloping curve since the amount of variability accounted for by successive p.c.’s are less and less.

I To determine the appropriate number of components, we look for a kink (or an elbow) in the scree plot, since it means that the curve flattens out after this and very little variability can be explained by the later principal components.

I The number of components is taken to be the point at which the remaining eigenvalues are relatively small and all about

(51)

Scree Plot

(52)

Scree Plot

(53)

Scree Plot

(54)

Scree Plot

(55)

Summary

I The principal components of standardized variables are looked into.

I The geometry of principal components is considered.

I Principal components of variables with some special covariance structures are discussed.

(56)

Summary

(57)

Summary