Geostatistics Lessons
Multivariate Statistics
Cite

Cite this lesson as:

Barnett, R. M. (2017). Principal Component Analysis. In J. L. Deutsch (Ed.), Geostatistics Lessons. Retrieved from https://geostatisticslessons.com/lessons/principalcomponentanalysis

Download PDF(1777KB) (opens in a new tab)

Principal Component Analysis

  • Ryan M. Barnett, University of Alberta

Learning Objectives

  • Understand principal component analysis (PCA) within the context of multivariate geostatistical modeling.
  • Review essential PCA theory relating to decorrelation and dimension reduction.
  • Interpret PCA results with data of varying dimensions to consolidate understanding of the technique.

Introduction

Principal component analysis (PCA) (Hotelling, 1933; Pearson, 1901) is a dimension reduction and decorrelation technique that transforms a correlated multivariate distribution into orthogonal linear combinations of the original variables. PCA is a useful geostatistical modeling tool for two primary reasons:

  1. Multivariate data, consisting of multiple correlated geological variables, are transformed by PCA to be uncorrelated. Independent geostatistical modeling of the decorrelated variables then proceeds, before the PCA back-transform restores the original correlation to the modeled variables.
  2. PCA may be used for dimension reduction in the above framework. Independent geostatistical modeling proceeds on a subset of the decorrelated variables, before the PCA back-transform provides models of all original variables.

PCA could be used to gain a deeper understanding of underlying latent factors, but in geostatistics these two reasons prevail. It was first applied to geostatistical modeling in this manner by (Davis & Greenes, 1983), with more recent examples from (Barnett & Deutsch, 2012) and (Boisvert, Rossi, Ehrig, & Deutsch, 2013). This lesson begins with a description of the data processing and covariance calculations that are necessary prior to applying PCA. Essential PCA theory is then outlined and demonstrated with a small example, before demonstrating it with a larger geochemical dataset.

Data Pre-processing and Covariance Calculation

Consider kk geological variables Z1,…,ZkZ_1,\ldots,Z_k that will be simulated across a stationary domain AA. Conditioning data is given as the matrix Z:zα,i,α=1,…,n,i=1,…,k\mathbf{Z}: z_{\alpha, i}, \alpha=1,\ldots,n, i=1,\ldots,k, where nn is the number of samples. The Z\mathbf{Z} data is assumed to be representative of the domain AA, so that parameters may be calculated experimentally. Variables must be transformed to have a mean of zero (termed centered) before applying PCA as formulated here. It is also recommended that the variables be transformed to have variance of one, as this improves the interpretability of subsequent PCA results. Standardization of the geological variables is therefore used as a pre-processor to PCA: Y:yα,i=(zα,i−μi)σi, for α=1,…,n,i=1,…,k\mathbf{Y}: y_{\alpha,i} = \frac{(z_{\alpha,i}- \mu_i)}{ \sigma_i}, \text{ for } \alpha=1,\ldots,n,i=1,\ldots,k

where μi=1/n∑α=1nzα,i\mu_i=1/n\sum_{\alpha=1}^n z_{\alpha,i} is the mean of ZiZ_i and σi2=1/n∑α=1nzα,i2−μi2\sigma_i^2 = 1/n\sum_{\alpha=1}^n z_{\alpha,i}^2 - \mu_i^2 is the variance of ZiZ_i. Each standardized YiY_i variable has a mean of zero and a variance of one. PCA revolves around the covariance matrix Σ\boldsymbol{\Sigma} of the Y\mathbf{Y} data, which is calculated as:
Σ:Ci,j=1n∑α=1nyα,i⋅yα,j, for i,j=1,…,k\boldsymbol{\Sigma}: C_{i,j}=\frac{1}{n} \sum_{\alpha=1}^n y_{\alpha,i} \cdot y_{\alpha,j}, \text{ for } i,j=1,\ldots,k

The Σ\boldsymbol{\Sigma} values parameterize the multivariate system of the Y\mathbf{Y} data in terms of linear variability and dependence. Diagonal entries Ci,iC_{i,i} are the variance of each YiY_i. Off-diagonal entries Ci,j,i≠jC_{i,j}, i \ne j are the covariance between YiY_i and YjY_j. These covariances are also correlations since each YiY_i has a variance of one.

PCA results are subject to the accuracy of Σ\boldsymbol{\Sigma}. If the calculated sample Σ\boldsymbol{\Sigma} is not representative of the true population covariances, then PCA will not make the population uncorrelated in reality. For example, the covariance calculation is very sensitive to outlier values. Careful exploratory analysis should be performed to detect and remove erroneous outliers from the data prior to the covariance calculation.

The familiar normal score transform may be considered in the place of standardization, as normal scores have a mean of zero, variance of one and no univariate outliers. This will likely improve robustness of the covariance calculation and linear rotation, although multivariate outliers may persist to have adverse consequences. The normal score transform is non-linear, which has implications for estimation. Back-transforming normal score estimates directly will introduce a bias, although transforming a series of quantiles, as in PostMG (Lyster & Deutsch, 2004), solves this issue.

PCA Transform

The first step of PCA performs spectral decomposition of Σ\boldsymbol{\Sigma}, yielding the eigenvector matrix V:vi,j,i,j=1,…,k\mathbf{V}:v_{i,j},i,j=1,\ldots,k and the diagonal eigenvalue matrix D:di,i,i=1,…,k\mathbf{D}:d_{i,i},i=1,\ldots,k: Σ=VDVT\boldsymbol{\Sigma} = \mathbf{VDV}^T

The PCA transform is then performed through the matrix multiplication of Y\mathbf{Y} and V\mathbf{V}: P=YV\mathbf{P}= \mathbf{YV}

This rotates the multivariate data so that the resultant principal components in P\mathbf{P} are uncorrelated. Multiplying P\mathbf{P} by the transpose of V\mathbf{V} rotates the data back to Y\mathbf{Y}, providing the back-transform that may be used for simulated realizations of the principal components: Y=PVT\mathbf{Y}= \mathbf{PV}^T

The eigenvector matrix may be thought of as a rotation matrix, providing a new basis where the correlated data are made orthogonal. The linear matrix multiplication of Y1,…,YkY_1,\ldots,Y_k with the ithi^{th} column of V\mathbf{V} provides the PiP_i principal component. Hence, each principal component is a linear combination of the original variables, explaining the nature of the linear rotation terminology.

Each di,id_{i,i} entry corresponds with the variance of PiP_i, while also measuring the variability that PiP_i explains about the Y1,…,YkY_1,\ldots,Y_k multivariate system. More specifically, the percentage variability that PiP_i explains about the Y1,…,YkY_1,\ldots,Y_k is calculated as di,i/∑j=1kdj,j⋅100d_{i,i}/\sum_{j=1}^kd_{j,j} \cdot 100, or di,i/tr(D)⋅100d_{i,i}/\textrm{tr}(\mathbf{D})\cdot 100. The component P1P_1 explains the most variability, P2P_2 explains the second most, and so on.

PCA is demonstrated using a small k=3k=3 example. The scatter plot of the Y1,…,Y3Y_1,\ldots,Y_3 data is overlain with the P1,…,P3P_1,\ldots,P_3 principal component vectors, which correspond with each column of V\mathbf{V} and display the rotation basis (e.g., axes of principal components). The vector lengths are scaled according to the associated eigenvalues, which are also displayed in the bar chart below.

Scatter plot of the original data with the orientation (eigenvector) and magnitude (eigenvalue) of the principal components overlain.
Figure 1: Scatter plot of the original data with the orientation (eigenvector) and magnitude (eigenvalue) of the principal components overlain.
Eigenvalues of each principal component.
Figure 2: Eigenvalues of each principal component.

Following transformation, the below scatter plot displays P\mathbf{P} data in the rotated basis, where the greatest variance exists visibly in the P1P_1 dimension. Scatter plots in this lesson are colored according to their associated Y3Y_3 value, which indicates how each data point is rotated and shifted by the various transformations. The nature of this PCA rotation may be understood by comparing the Y\mathbf{Y} and P\mathbf{P} scatter plots. The displayed covariance matrices confirm that: 1) the data are made uncorrelated according to the off-diagonal entries and 2) the principal components contain decreasing variance according to eigenvalues in the diagonal entries.

Scatter plot of the PCA data.
Figure 3: Scatter plot of the PCA data.
Covariance matrix of the original data (left) and PCA data (right).
Figure 4: Covariance matrix of the original data (left) and PCA data (right).

Dimension Reduction

Since eigenvalues measure the variability that each principal component contributes to the original multivariate system, practitioners may consider discarding insignificant components from subsequent geostatistical modeling. This is not utilized often in practice, but is available when an infeasibly large number of variables must be modeled. Consider that the ll most important principal components are selected for simulation across NN model nodes, where l<kl < k. Letting the resulting realization values be the N×lN\times l matrix P′\mathbf{P}', the PCA back-transform is simply modified by multiplying P′\mathbf{P}' with the first ll rows of VT\mathbf{V}^T. The multiplication of these N×lN\times l and l×kl\times k matrices yields the N×kN\times k matrix Y\mathbf{Y} of the standardized variable realizations.

The effectiveness of this dimension reduction scheme relates to the magnitude of variance that the removed principal components explain. If the associated eigenvalues are vanishingly small, then removal of those principal components should not have a significant impact on simulation results. The figure below demonstrates the PCA back-transform of l=1l=1 and l=2l=2 of the k=3k=3 components. Rather than a simulated realization, the transformed P\mathbf{P} data is simply being back-transformed. The true standardized data values are compared with the back-transformed values, where perfect reproduction of the original values would be achieved if back-transforming all k=3k=3 components, leading to all scatter falling on a 45 degree line and correlation of ρ=1\rho=1. Scatter about the 45 degree line represents imperfect reproduction of the data, resulting from the loss of variability that would have been explained by the removed principal component(s). In this case, each principal component explains a significant amount of variability, so that the impact of their removal is substantial. Smaller eigenvalues can be expected as kk grows larger, making the use of dimension reduction more effective.

Scatter between the true and back-transformed values using two (above) and one (below) principal components.
Figure 5: Scatter between the true and back-transformed values using two (above) and one (below) principal components.

It is interesting to note that Y1Y_1 reproduction is virtually identical whether using l=1l=1 or l=2l=2 principal components, whereas the reproduction of Y2Y_2 and Y3Y_3 is significantly improved. This relates to the nature of the rotation and how the original variables are loaded onto the principal components. A loading ρ′(Yi,Pj)\rho'(Y_i,P_j) describes how important the PjP_j principal component is for characterizing the YiY_i variability. It is calculated as: ρ′(Yi,Pj)=vi,j⋅dj,j=ρ(Yi,Pj)⋅σYi\rho'(Y_i ,P_j ) = v_{i,j} \cdot \sqrt{d_{j,j}} = \rho (Y_i ,P_j ) \cdot \sigma_{Y_i}

This shows that a loading is the product of eigenvectors vi,jv_{i,j} and the square roots of eigenvalues dj,jd_{j,j}, though it may be more intuitively thought of as the correlation ρ\rho between the original and transformed variables, scaled by the standard deviation σYi\sigma_{Y_i} of YiY_i. When working with standardized data, as we are here, a loading is simply the correlation between the YiY_i original variable and the PjP_j principal component, ρ′(Yi,Pj)=ρ(Yi,Pj)\rho'(Y_i,P_j) = \rho(Y_i ,P_j ). Inspecting the loadings of this transformation, observe that P2P_2 is virtually uncorrelated with Y1Y_1. That is why the results above show that the inclusion and exclusion of P2P_2 in the back-transform yields virtually identical results for Y1Y_1. All of the original variables are loaded most heavily on P1P_1, which is expected since it explains the majority of their variability.

Loadings of the standardized variables on the principal components.
Figure 6: Loadings of the standardized variables on the principal components.

Geochemical Example

A geochemical dataset provides a more compelling example of PCA in terms of potential dimension reduction and exploratory analysis. This public data was collected by the Northwest Territories Geological Survey in partnership with the Geological Survey of Canada. It includes n=1660n=1660 stream sediment samples that provide k=53k=53 elements, which were collected in mineral deposit exploration across the Mackenzie Mountains. After standardizing the elements, the covariance matrix of the resulting Y\mathbf{Y} data is calculated and displayed below.

Covariance matrix of the standardized elements.
Figure 7: Covariance matrix of the standardized elements.

Spectral decomposition is applied to the covariance matrix, generating the eigenvalues displayed below. The explained variability of the principal components is then calculated from the eigenvalues, which is displayed in an incremental and cumulative manner. The eigenvalue plot is sometimes referred to as a scree or elbow plot. It is a useful tool, particularly if a visible elbow or inflection exists, where principal components begin explaining insignificant variability. A slight elbow exists here after the third or fourth component, though users may consider modeling additional components based on a required threshold of explained variability; say the 29 components that are required here to explain 95% of the variability.

Eigenvalues and explained variability of each principal component.
Figure 8: Eigenvalues and explained variability of each principal component.

As expected based on the eigenvalues, most elements are only loaded strongly onto the first few principal components (matrix below). Consider that the volume of information in the covariance and loadings matrices above creates challenges for interpreting the overall multivariate system. A common exploratory analysis approach for simplifying the multivariate system and understanding the underlying latent variables, involves plotting the loadings of select principal components against each other, such as the first two principal components that are plotted below. Elements located in close proximity are closely related, and vice versa. For example, consider that Ca and Mg have the largest P1 values (furthest right on the plot), while having very small P2 values. Their variability is largely explained by P1, but not P2. The two elements are located very near to each other and relatively far from other elements, which corresponds with the covariance matrix, where Ca and Mg are highly correlated with each other, and strongly negatively correlated with most of the other elements. As only the first two principal component loadings are displayed, note that this is a simplified projection of the multivariate system, which only explains 49% of the variability.

Loadings of the standardized elements on the principal components.
Figure 9: Loadings of the standardized elements on the principal components.
Scatter of loadings with the first two principal components.
Figure 10: Scatter of loadings with the first two principal components.

Summary

PCA is a useful tool for multivariate geostatistical modeling. Geological variables are decorrelated to facilitate independent modeling, before the back-transform restores the original correlation to modeled variables. When the number of variables becomes impractical to model, the dimension reduction functionality of PCA may be used for modeling a subset of variables, before the back-transform provides models of all variables. It may also be applied for exploratory data analysis, providing insight into the underlying latent variables that explain a high dimensional multivariate system.

There are alternative linear decorrelation transformations that are immediate extensions of PCA, which may offer advantages to geostatistical modeling. These include data sphereing and minimum/maximum autocorrelation factors, which are the focus of a companion lesson.

References

Barnett, R. M., & Deutsch, C. V. (2012). Practical implementation of non-linear transforms for modeling geometallurgical variables. In P. Abrahamsen, R. Hauge, & O. Kolbjornsen (Eds.), Geostatistics Oslo 2012 (pp. 409–422). Springer, Netherlands.
Boisvert, J. B., Rossi, M. E., Ehrig, K., & Deutsch, C. V. (2013). Geometallurgical modeling at Olympic Dam Mine, South Australia. Mathematical Geosciences, 45, 901–925.
Davis, B. M., & Greenes, K. A. (1983). Estimation using spatially distributed multivariate data: An example with coal quality. Mathematical Geology, 15, 287–300.
Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24, 417–441.
Lyster, S., & Deutsch, C. V. (2004). PostMG: A postprocessing program for multiGaussian kriging output (CCG Annual Report 6 No. 405). University of Alberta, Edmonton, Canada: University of Alberta.
Pearson, K. (1901). On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11), 559–572.

Cite this lesson as:

Barnett, R. M. (2017). Principal Component Analysis. In J. L. Deutsch (Ed.), Geostatistics Lessons. Retrieved from https://geostatisticslessons.com/lessons/principalcomponentanalysis

Search lesson titles and content.