With the advancements within the field of Machine Learning and computer science, it’s become essential to grasp the basics behind such technologies. This blog on Principal Component Analysis will facilitate your understanding of the concepts behind dimensionality reduction and the way it may be wont to house high-dimensional data.<\/p>\n
The Below mentioned Principal Component Analysis Tutorial will help to Understand the detailed information about what is PCA in machine learning, so Just follow all the tutorials of India\u2019s Leading Best Data Science Training institute in Bangalore<\/a> and Be a Pro Data Scientist<\/a> or Machine Learning Engineer.<\/p>\n Suppose you have to deal with the dataset based on GDP (Gross Domestic Production) of any country.\u00a0 While processing on data you will come across so many fields, which we call as features or variables. Now the question arises, how we can take all variables and select only a few of them for further processing? There may be a problem of overfitting the model. So we have to reduce the dimensionality of that data so that it will be easier and we can have a smaller amount of relationships between variables to consider. So, reducing dimensions of feature space is nothing but \u2018Dimensionality Reduction.<\/p>\n There are different methods to achieve dimensionality reduction. Mostly used methods are categorized into:<\/p>\n Features Elimination<\/p>\n Features Extraction<\/p>\n As the name suggests, in this method the features are eliminated which are having very low significance. Here only the best features are selected as per domain requirement. The rest features are eliminated. Simplicity and maintaining interpretability are the advantages of variables in this method.<\/p>\n Suppose we have N number of independent variables. In feature extraction, we create N \u201cnew\u201d independent variables, where each newly generated independent variable is a combination of each of the N \u201cold\u201d independent variables. However, the newly created variables have arranged in a specific order based on how well they predict the dependent variable. While following these methods, we always eliminate or remove the features which are \u2018least important\u2019. And there \u2018Dimensionality reduction\u2019 comes in a picture. As we ordered the new variables by how well they predict our dependent variable, we are well known about which variable is the most important and least important. Still, we are keeping the most valuable parts of our old variables, even when we drop one or more of these \u201cnew\u201d variables.<\/p>\n \u00a0<\/strong><\/p>\n If we want to reduce the number of variables, but we are unable to identify which variables to completely remove from consideration then PCA is the best option. Even, there are some cases where we can make independent variables less interpretable and we have to ensure those variables are independent of each other also.<\/p>\n Principal Component Analysis (PCA) is a numerical procedure that uses an orthogonal alteration. It converts a set of correlated variables to a set of uncorrelated variables. Investigative data analysis and predictive models use PCA as an effective tool. It is also called general factor analysis, as a line of best fit is determined by regression.<\/p>\n In simple words, PCA takes a dataset with a lot of dimensions and flattens it to 2 or 3 dimensions. It tries to find a meaningful way to flatten the data by focusing on the things that are different between independent variables.<\/p>\n The image above shows the example of a transformation of high-dimensional data i.e. 3-dimensional data to low dimensional i.e. 2-dimensional data using PCA. Before moving to the actual concept, let\u2019s see some terminologies related to PCA.<\/p>\n Dimensionality:<\/strong> \u00a0It is simply the number of features or the number of columns present in our dataset. We can consider it as a number of random variables in a dataset<\/p>\n Correlation:<\/strong> \u00a0It displays how strongly two variables are related to each other. The value ranges from -1 to +1. Positive indicates that if one variable increases, the other will increase as well, while negative indicates the other decreases on increasing the other. And the modulus value indicates the strength of the relation.<\/p>\n Orthogonal:<\/strong> \u00a0Uncorrelated to every other, i.e., a correlation between any pair of variables is 0.<\/p>\n Eigenvectors:<\/strong> \u00a0Let\u2019s consider a non-zero vector v. Let\u2019s take a square matrix A. So it is an eigenvector of a A, if Av is a scalar multiple of v. Or simply:<\/p>\n Av = \u019bv<\/p>\n Here, v is the eigenvector<\/p>\n \u019b is the eigenvalue associated with it.<\/p>\n Covariance Matrix:<\/strong> This matrix consists of covariance between the pairs of variables. The covariance between i-th and j-th variable is nothing but this (i,j)th element.<\/p>\n A normalized linear combination of the original predictors in a data set is called a principal component is. In the image above, the principal components are indicated by PC1<\/em>\u00a0and\u00a0PC2<\/em>. We can fit the data into two axes, which are nothing but these principle components i.e. PCs.<\/p>\n PC1 is the first principle axis that spans the most variation, whereas PC2 is the second principal axis which spans the second most variation. Means PC1 will capture the directions where most of the variation is present and PC2 captures the direction with the second-most variation.<\/p>\n The PCs are essentially the linear combinations of the original variables, the weights vector in this combination is actually the eigenvector found which in turn satisfies the principle of least squares.<\/p>\n The PCs are orthogonal in nature.<\/p>\n As we move from the 1st PC to the last one, the variation present in PCs decreases.<\/p>\n Sometimes in regression, outlier detection problems, these least important PCs are useful.<\/p>\n Implementing PCA on a 2-D Dataset<\/p>\n The first step is to normalize data that we have so that PCA works properly. This is done by subtracting respective means from numbers in the respective column. Let\u2019s consider we have two dimensions X and Y. All X will be ?- and all Y will be ?-. This produces a dataset whose mean is zero.<\/p>\n Since the dataset we took is 2-dimensional, this will give result in a 2×2 Covariance matrix.<\/p>\n Please note that<\/p>\n Var[X1] = Cov[X1,X1]<\/p>\n Var[X2] = Cov[X2,X2].<\/p>\n The next step is to calculate eigenvalues and eigenvectors for the covariance matrix. For a matrix A, \u019b is an eigenvalue which is a solution of the characteristic equation:<\/p>\n det( \u019bI – A ) = 0<\/p>\n Where,<\/p>\n It is the identity matrix of the same dimension as matrix A. It is a required condition for matrix subtraction.<\/p>\n \u2018det\u2019 is the determinant of the matrix.<\/p>\n For each eigenvalue \u019b, a corresponding eigenvector v can be found by solving:<\/p>\n ( \u019bI – A )v = 0<\/p>\n We order eigenvalues from largest to smallest so that it gives us components in order or significance. Here comes the dimensionality reduction part. If we have data with n variables, then we have corresponding n eigenvalues and eigenvectors. It turns out that the eigenvector corresponding to the highest eigenvalue is the principal component of the dataset and it is our call as to how many eigenvalues we choose to move ahead of our analysis with. To reduce dimensions, we choose the first p eigenvalues and ignore rest. We do lose out some information in the process, but if eigenvalues are small, we do not lose much.<\/p>\n Next, we will form a feature vector which is a matrix. This is a matrix of vectors with the eigenvectors which we want to proceed with. Since we have just 2 dimensions in the running example, we can either choose one corresponding to greater eigenvalue or simply take both.<\/p>\n Feature Vector = (eig1, eig2)<\/p>\n This is the final step where we actually form principal components using all math we did till here. For the same, we take the transpose of the feature vector and left-multiply it with the transpose of a scaled version of the original dataset.<\/p>\n Here,<\/p>\n NewData = Matrix consisting of the principal components,<\/p>\n Feature Vector = matrix we formed using eigenvectors we chose to keep, and<\/p>\n Scaled Data is a scaled version of the original dataset<\/p>\n Where T denotes the transpose of matrices.<\/p>\n If we go back to the theory of eigenvalues and eigenvectors, we will see that, essentially, eigenvectors provide us with information about patterns in data. In this example, if we plot eigenvectors on a scatterplot of data, we find that the principal eigenvector actually fits well with data. The other one, being perpendicular to it, does not carry much information, and hence, we are at not at much loss when deprecating it, hence reducing the dimension.<\/p>\n All the eigenvectors of a matrix are orthogonal i.e. perpendicular to each other. So, in PCA, what we do is represents or transforms the original dataset using these orthogonal (perpendicular) eigenvectors instead of representing on the normal x and y\u00a0 axis. We have now classified our data points as a combination of contributions from both x and y. The difference lies when we actually disregard one or many eigenvectors, hence, reducing the dimension of the dataset. Otherwise, in case, we take all eigenvectors into an account, we are just transforming coordinates and hence, not serving a purpose.<\/p>\n PCA is predominantly used as a type of a dimensionality reduction technique in domains like facial recognition, computer vision, and image compression. It is also used for determining patterns in data of high dimension in fields of finance, data mining, bioinformatics, psychology, etc.<\/p>\n %matplotlib inline<\/p>\n from sklearn.datasets import load_breast_cancer<\/p>\n cancer=load_breast_cancer()<\/p>\n df=pd.DataFrame(cancer[‘data’],columns=cancer[‘feature_names’])<\/p>\n df.head()<\/p>\n from sklearn.preprocessing import StandardScaler<\/p>\n scl= StandardScaler()<\/p>\n scl.fit(df)<\/p>\n scl_data=scl.transform(df)<\/p>\n from sklearn.decomposition import PCA<\/p>\n pca=PCA(n_components=2)<\/p>\n pca.fit(scl_data)<\/p>\n x_pca=pca.transform(scl_data)<\/p>\n x_pca.shape<\/p>\n #a larger plot<\/p>\n plt.figure(figsize=(8,6))<\/p>\n plt.scatter (x_pca[:,0],x_pca[:,1],c=cancer[‘target’],cmap=’viridis’)<\/p>\n # Labeling to axes<\/p>\n plt.xlabel(‘First Principle component’)<\/p>\n plt.ylabel(‘Second Principle Component’)<\/p>\n Output:<\/strong><\/p>\n pca.components_<\/p>\n Output:<\/strong><\/p>\n We hope you understand Principal Component Analysis Tutorial. Get success in your career as a Data Scientist by being a part of the Prwatech<\/a>, India’s leading Data Science training institute in Bangalore<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":" Principal Component Analysis Tutorial Principal Component Analysis Tutorial, in this Tutorial one, can learn types of principal component analysis. Are you the one who is looking for the best platform which provides information about know working principle of PCA, Applications of Principal Component Analysis? Or the one who is looking forward to taking the advanced […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[32,1696],"tags":[463,464,459,461,460,462],"class_list":["post-4950","post","type-post","status-publish","format-standard","hentry","category-machine-learning","category-machine-learning-modules","tag-applications-of-principal-component-analysis","tag-implementation-of-pca-using-python","tag-principal-component-analysis-tutorial","tag-types-of-principal-component-analysis","tag-what-is-pca-in-machine-learning","tag-working-principle-of-pca"],"acf":[],"yoast_head":"\nWhat is PCA in Machine Learning?<\/h2>\n
Types of Principal Component Analysis<\/h3>\n
Feature Elimination<\/h4>\n
Feature Extraction<\/h4>\n
When PCA can be used?<\/h3>\n
Working Principle of PCA<\/h2>\n
![\"Working](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/02\/PCA-Working-principle.png\")
<\/p>\nPrincipal Components<\/h3>\n
Step 1: Normalize the data:<\/h3>\n
Step 2: Calculate the covariance matrix<\/h3>\n
Step 3: Calculate the eigenvalues and eigenvectors:<\/h3>\n
Step 4: Selecting components and forming a feature vector:<\/h3>\n
Step 5: Forming Principal Components<\/h3>\n
![\"Principal](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/02\/Forming-principle-componens-formula.png\")
<\/p>\nApplications of Principal Component Analysis<\/h3>\n
Step by Step Implementation of PCA using Python<\/h3>\n
Import required libraries<\/h4>\n
Import dataset<\/h4>\n
![\"Step](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/02\/Dataset-1024x249.png\")
<\/p>\nNormalize DataSet using Standard Scalar<\/h4>\n
Import and implement PCA<\/h4>\n
![\"Principal](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/02\/import-n-implement-PCA-oputput.png\")
<\/p>\nDisplay Dimension reduced<\/h3>\n
![\"Principal](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/02\/PCA.png\")
<\/p>\n