7.8: PCA

Principal component analysis (PCA) can help us with dimensionality reduction. It is a feature extraction technique. It can help us reduce a huge set of highly correlated features into a smaller set of 'components'. These 'components' are uncorrelated from each other. It has a very specific application in removing multicollinearity from features in linear models. One caveat with PCA is that the principal components are not ordered based on their importance for the supervised or unsupervised machine learning problem at hand.

Hence, although it might appear intuitive to perform feature selection on these reduced principal components, it is not recommended. If the model performance improves as a result of the feature selection of PCA components, it will be pure coincidence.