k-Nearest Neighbors Algorithm Tutorial

  • date 15th February, 2020 |
  • by Prwatech |


k-Nearest Neighbors Algorithm Tutorial


k-nearest neighbors Algorithm Tutorial, Are you the one who is looking forward to know about knn algorithm introduction in Machine Learning? Or the one who is looking forward to know How KNN algorithm works and benefits of knn or Are you dreaming to become to certified Pro Machine Learning Engineer or Data Scientist, then stop just dreaming, get your Data Science certification course with Machine Learning from India’s Leading Data Science training institute.


Knn is part of supervised learning which will be used in many applications such as data mining, image processing and many more. It is simple and one of the most important Machine learning algorithms. In this blog, we will learn knn algorithm introduction, knn implementation in python and benefits of knn. Do you want to know How KNN algorithm works, So follow the below mentioned k-nearest neighbors algorithm tutorial from Prwatech and take advanced Data Science training with Machine Learning like a pro from today itself under 10+ Years of hands-on experienced Professionals.


knn Algorithm Introduction


K nearest Neighbor (KNN) is a popular supervised machine learning algorithm that is used widely. Its popularity stems from its comfort of use, and its clearly reasonable results. The ‘K’ in KNN indicates the number of nearest neighbors, which are used to classify or predict outputs in a data set. The classification or prediction of every new data point is based on a specific distance calculated from nearest neighbors and their weighted averages.


If there is very less prior knowledge about how the data points are distributed, then KNN is best algorithm for analysis.In case of labeled datasets, KNN helps to find the relationship between each observation in training data.This relationship of resemblance is expressed by means of a distance between data points.There are different ways to calculate the intervals between observations like Euclidean, Manhattan, Chebyshev, Hamming distance etc.


KNN is a non-parametric algorithm which makes no clear assumptions about the functional form of the relationship. Rather it works directly on training instances than applying any specific model.KNN can be used to solve prediction problems based on both classification and regression. But it is more widely used for prediction in classification problems in the industry.


How KNN algorithm works?


As mentioned above KNN basically works on the relationship of resemblance by means of calculating distance of data point from others.

Let’s take an example.


k-nearest neighbors Algorithm Tutorial


Let’s consider a set of some data points is given which includes red stars and blue squares. Now we have to categorize a new data point which is indicated with green circle. For applying KNN, first we have to decide value of K. Let’s consider value of K be 3. Now based on K=3 we have to find 3 neighbors which are nearest to this green circle.


k-nearest neighbors Algorithm Tutorial


In this example the green circle has to find its nearest neighbors. And as we can see all the red stars are nearest to the circle compared to blue squares. So, the new data point i.e. green circle will be classified as ‘red star’.


To get best possible fit for the data set, it is important to select the appropriate value of ‘K’ i.e. number of nearest neighbors. If K value is small, the region of prediction is restricted and it makes the classifier less sensitive the overall distribution. K should be most common class value in case of classification and it should be mean output variable in case of regression.


Methods of calculating distance between data points:


To find the distance of target data point means to measure the interval between new data point and each data point present around it.  To calculate this the most commonly used methods are: Euclidian, Manhattan and Hamming distance.


Euclidian Distance:


It is the square root of the sum of the squared differences between a new point and an existing point. Let’s take new point as x and existing point y.


Euclidian Distance


Manhattan Distance:


It is the distance between real vectors using the sum of their absolute difference.

Manhattan Distance


Hamming Distance:


Hamming distance is helpful in case of categorical variables. If the x value and y value are same, the distance will be zero. Otherwise distance is:

Here distance is indicated with ‘D’.


Hamming Distance


How to decide K value?


The most important step in KNN is to determine the optimal value of K. The optimal value of k decreases effect of the noise on the classification. For this a technique called ‘Elbow method’ helps to select the optimal K value.


Let’s say we are applying K with different values to same dataset. Initially we have to observe the error rate changing with K values. Let’s say we applied KNN on a data set and we got a curve of error rate Vs. K values as follows:


How to decide K value?


Here we can observe that initially error rate is decreasing up to nearly 6. After that it starts increasing. So, the value of K should be 6 i.e. it is optimum value of K for this model. This curve is called because it has a shape like an elbow and is usually used to regulate the k value. This is called Elbow method.


knn implementation in python


In the following example we will consider k=3, hence, we will take 3 nearest points to new or target point.


Importing Library


import numpy as np

import pandas as pd

import matplotlib.pyplot as plt


Importing dataset


df= pd.read_csv(“ Your File Path”)

(We will consider that necessary cleaning part is done)


Divide the dataset into independent and dependent parts.


X = data.iloc[:,0:20]  #independent columns

y = data.iloc[:,-1] # target column



Split the dataset into two parts for training and testing.


from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 10, test_size = 0.33)


Preprocessing – Scaling the features


from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler(feature_range=(0, 1))

x_train_scaled = scaler.fit_transform(x_train)

x_train = pd.DataFrame(x_train_scaled)

x_test_scaled = scaler.fit_transform(x_test)

x_test = pd.DataFrame(x_test_scaled)


Visualize the error rate for different k values


#import required packages

from sklearn import neighbors

from sklearn.metrics import mean_squared_error

from math import sqrt

import matplotlib.pyplot as plt

%matplotlib inline

rmse_val = []


#to store rmse values for different k

for K in range(20):

K = K+1

model = neighbors.KNeighborsRegressor(n_neighbors = K)

model.fit(x_train, y_train)           #fit the model

pred=model.predict(x_test)          #make prediction on test set

error = sqrt(mean_squared_error(y_test,pred))


#calculate rmse

rmse_val.append(error) #store rmse values

print(‘RMSE value for : ‘ , K , ‘is:’, error)


#plotting the rmse values against k values

curve = pd.DataFrame(rmse_val)     #elbow curve



knn implementation in python


Implementing GridsearchCV


Plotting the elbow curve every time to determine K, is be a cumbersome and tedious process. You can simply use gridsearch to find the best value.


from sklearn.model_selection import GridSearchCV

params = {‘n_neighbors’:[2,3,4,5,6,7,8,9]}

knn = neighbors.KNeighborsRegressor()

model = GridSearchCV(knn, params, cv=5)




Benefits of knn:


The algorithm is useful in searching for similar documents. This algorithm can be used to find all the e-mails, briefs, contracts, etc. relevant to given problem statements. Recommendation systems prefer this algorithm to recommend customer different products based on past purchasing history. Also, it is used as a benchmark for more complex classifiers such as Artificial Neural Networks (ANN).


We hope you understand k-nearest neighbors algorithm tutorial.Get success in your career as a Data Scientist by being a part of the Prwatech, India’s leading Data Science training institute in Bangalore.


Quick Support

image image