{"id":4831,"date":"2020-02-15T11:57:39","date_gmt":"2020-02-15T11:57:39","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=4831"},"modified":"2024-03-15T11:43:38","modified_gmt":"2024-03-15T11:43:38","slug":"boosting-techniques-in-machine-learning","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/boosting-techniques-in-machine-learning\/","title":{"rendered":"Boosting Techniques in Machine Learning"},"content":{"rendered":"

Boosting Techniques in Machine Learning<\/h1>\n

Boosting Techniques in Machine Learning<\/strong>, in this Tutorial one can learn the Boosting algorithm introduction. Are you the one who is looking for the best platform which provides information about different types of boosting algorithm? Or the one who is looking forward to taking the advanced Data Science Certification Course<\/a> from India\u2019s Leading Data Science Training institute<\/a>? Then you\u2019ve landed on the Right Path.<\/p>\n

Machine learning is vital for many of the technologies that seek to provide intelligence to the data, and companies recognize the great value. Boosting is a meta-algorithm joint learning machine to mainly reduce bias, and also variation in supervised learning, and a family of machine learning algorithms that convert students’ weaknesses to strengths.<\/p>\n

The Below mentioned Tutorial will help to Understand the detailed information about boosting techniques in machine learning, so Just follow all the tutorials of India\u2019s Leading Best Data Science Training institute in Bangalore<\/a> and Be a Pro Data Scientist<\/a> or Machine Learning Engineer.<\/p>\n

Boosting Algorithm Introduction<\/h3>\n

As seen in the introduction part of ensemble methods, boosting is one of the advanced ensemble methods which improve overall performance by decreasing bias. Boosting is a consecutive process, where each succeeding model attempts to correct the errors of the preceding model.<\/p>\n

Different Types of Boosting Algorithm<\/h2>\n

There are mainly five types of boosting techniques.<\/p>\n

AdaBoost<\/p>\n

Gradient Boosting (GBM)<\/p>\n

Light GBM<\/p>\n

XGBoost<\/p>\n

CatBoost<\/p>\n

Let\u2019s see more about these types.<\/p>\n

AdaBoost Algorithm in Machine Learning<\/h3>\n

Understanding AdaBoost Algorithm<\/strong><\/p>\n

AdaBoost, considered one of the simplest boosting algorithms, commonly utilizes decision trees for modeling. It follows a sequential approach, wherein multiple models are created successively. Each subsequent model aims to rectify the errors made by its predecessor.<\/p>\n

This correction process involves assigning higher weights to observations that were incorrectly predicted by the previous model. Consequently, in the subsequent iteration, the algorithm prioritizes the accurate prediction of these misclassified instances. This iterative refinement of models ensures progressive improvement in overall prediction accuracy.<\/p>\n

:<\/p>\n

\"Boosting<\/p>\n

where<\/p>\n

fm\u00a0<\/sub>= mth<\/sup> weak classifier<\/p>\n

m<\/sub>= corresponding weight.<\/p>\n

Understanding AdaBoost Algorithm<\/strong><\/p>\n

AdaBoost, short for Adaptive Boosting, is precisely the weighted combination of multiple weak classifiers. The algorithm operates through the following steps:<\/p>\n

    \n
  1. Equal Weight Initialization:<\/strong> Initially, all data points in the dataset are assigned equal weights.<\/li>\n
  2. Model Building:<\/strong> A subset of the data is utilized to build a weak model.<\/li>\n
  3. Prediction:<\/strong> Predictions are made for the entire dataset based on this model.<\/li>\n
  4. Error Measurement:<\/strong> Errors are evaluated by comparing the predictions with the actual values.<\/li>\n
  5. Weight Adjustment:<\/strong> Data points that are incorrectly predicted are assigned higher weights in the creation of the next model.<\/li>\n
  6. Weight Calculation:<\/strong> The weights for data points are determined based on the magnitude of errors. Higher errors correspond to higher weights for the respective data points.<\/li>\n
  7. Iterative Process:<\/strong> This process is repeated until either the error function stabilizes or the maximum limit of estimators is reached.<\/li>\n<\/ol>\n

    Let’s illustrate this process with an example of employee attrition.<\/p>\n

    Initializing and importing libraries<\/h4>\n

    Import pandas as pd<\/p>\n

    Import numpy as np<\/p>\n

    Reading File<\/h4>\n

    df=pd.read_csv(“Your File Path”<\/em>)<\/p>\n

    df.head()<\/p>\n

    Splitting dataset into train and test<\/h4>\n

    from sklearn.model_selection import train_test_split<\/p>\n

    train, test = train_test_split(df, test_size=0.3, random_state=0)<\/p>\n

    x_train=train.drop(‘status’,axis=1)<\/p>\n

    y_train=train[‘status’]<\/p>\n

    x_test=test.drop(‘status’,axis=1)<\/p>\n

    y_test=test[‘status’]<\/p>\n

    Applying AdaBoost:<\/h4>\n

    from sklearn.ensemble import AdaBoostClassifier<\/p>\n

    model = AdaBoostClassifier(random_state=1)<\/p>\n

    model.fit(x_train, y_train)<\/p>\n

    model.score(x_test,y_test)<\/p>\n

    Output:<\/strong><\/h4>\n

    0.9995555555555555<\/p>\n

    Note: In the case of regression, the steps will be the same. Just we have to replace AdaBoostClassifier with AdaBoostRegressor.<\/p>\n

    \u00a0<\/strong><\/p>\n

    Gradient Boosting (GBM) in Machine Learning:<\/h3>\n

    Gradient Boosting, or GBM, represents an ensemble machine learning algorithm applicable to both regression and classification problems. GBM combines numerous weak learners to construct a robust algorithm. Similar to other boosting methods, each subsequent tree in GBM is built based on the error calculation from the preceding trees. In GBM, regression trees serve as the base learner.<\/p>\n\n\n\n\n\n\n\n\n\n\n\n
    Gender<\/strong><\/td>\nHeight<\/strong><\/td>\nWeight<\/strong><\/td>\nBMI<\/strong><\/td>\nPhysical Activity<\/strong><\/td>\nAge<\/strong><\/td>\n<\/tr>\n
    M<\/strong><\/td>\n160<\/td>\n85<\/td>\n33.20<\/td>\n1<\/td>\n35<\/td>\n<\/tr>\n
    F<\/strong><\/td>\n155<\/td>\n64<\/td>\n26.64<\/td>\n0<\/td>\n27<\/td>\n<\/tr>\n
    M<\/strong><\/td>\n170.7<\/td>\n95<\/td>\n28.14<\/td>\n0<\/td>\n28<\/td>\n<\/tr>\n
    F<\/strong><\/td>\n185.4<\/td>\n65<\/td>\n23.27<\/td>\n1<\/td>\n28<\/td>\n<\/tr>\n
    F<\/strong><\/td>\n158<\/td>\n70<\/td>\n36.05<\/td>\n1<\/td>\n32<\/td>\n<\/tr>\n
    F<\/strong><\/td>\n155<\/td>\n90<\/td>\n35.38<\/td>\n0<\/td>\n28<\/td>\n<\/tr>\n
    M<\/strong><\/td>\n173.7<\/td>\n72<\/td>\n23.86<\/td>\n1<\/td>\n22<\/td>\n<\/tr>\n
    F<\/strong><\/td>\n161.5<\/td>\n74<\/td>\n23.77<\/td>\n0<\/td>\n33<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    The mean age is supposed to be the predicted value (which is indicated by\u2018Predicition1\u2019) for all observations in the dataset. The difference between mean age and actual values of age is considered as errors, which is indicated by \u2018 Error 1\u2019.<\/p>\n

    Prediction 1:<\/h4>\n\n\n\n\n\n\n\n\n\n\n\n
    Gender<\/strong><\/td>\nHeight<\/strong><\/td>\nWeight<\/strong><\/td>\nBMI<\/strong><\/td>\nPhysical Activity<\/strong><\/td>\nAge<\/strong><\/td>\nMean Age<\/strong><\/td>\nError<\/strong><\/td>\n<\/tr>\n
    M<\/strong><\/td>\n160<\/td>\n85<\/td>\n33.20<\/td>\n1<\/td>\n35<\/td>\n29<\/td>\n6<\/td>\n<\/tr>\n
    F<\/strong><\/td>\n155<\/td>\n64<\/td>\n26.64<\/td>\n0<\/td>\n27<\/td>\n29<\/td>\n-2<\/td>\n<\/tr>\n
    M<\/strong><\/td>\n170.7<\/td>\n95<\/td>\n28.14<\/td>\n0<\/td>\n28<\/td>\n29<\/td>\n-1<\/td>\n<\/tr>\n
    F<\/strong><\/td>\n185.4<\/td>\n65<\/td>\n23.27<\/td>\n1<\/td>\n28<\/td>\n29<\/td>\n-1<\/td>\n<\/tr>\n
    F<\/strong><\/td>\n158<\/td>\n70<\/td>\n36.05<\/td>\n1<\/td>\n32<\/td>\n29<\/td>\n3<\/td>\n<\/tr>\n
    F<\/strong><\/td>\n155<\/td>\n90<\/td>\n35.38<\/td>\n0<\/td>\n28<\/td>\n29<\/td>\n-1<\/td>\n<\/tr>\n
    M<\/strong><\/td>\n173.7<\/td>\n72<\/td>\n23.86<\/td>\n1<\/td>\n22<\/td>\n29<\/td>\n-7<\/td>\n<\/tr>\n
    F<\/strong><\/td>\n161.5<\/td>\n74<\/td>\n23.77<\/td>\n0<\/td>\n33<\/td>\n29<\/td>\n4<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Using Error 1 calculation, a tree model is designed. The purpose is to reduce that error to 0.<\/p>\n

    Prediction 2:<\/h4>\n\n\n\n\n\n\n\n\n\n\n\n
    Gender<\/strong><\/td>\nPhysical Activity<\/strong><\/td>\nAge<\/strong><\/td>\nMean Age(prediction 1)<\/b><\/td>\nError1<\/strong><\/td>\nPrediction 2<\/strong><\/td>\nMean + prediction 2\u00a0<\/b><\/td>\n<\/tr>\n
    M<\/strong><\/td>\n1<\/td>\n35<\/td>\n29<\/td>\n6<\/td>\n4<\/td>\n33<\/td>\n<\/tr>\n
    F<\/strong><\/td>\n0<\/td>\n27<\/td>\n29<\/td>\n-2<\/td>\n-1<\/td>\n28<\/td>\n<\/tr>\n
    M<\/strong><\/td>\n0<\/td>\n28<\/td>\n29<\/td>\n-1<\/td>\n-1<\/td>\n29<\/td>\n<\/tr>\n
    F<\/strong><\/td>\n1<\/td>\n28<\/td>\n29<\/td>\n-1<\/td>\n0<\/td>\n29<\/td>\n<\/tr>\n
    F<\/strong><\/td>\n1<\/td>\n32<\/td>\n29<\/td>\n3<\/td>\n1<\/td>\n30<\/td>\n<\/tr>\n
    F<\/strong><\/td>\n0<\/td>\n28<\/td>\n29<\/td>\n-1<\/td>\n1<\/td>\n30<\/td>\n<\/tr>\n
    M<\/strong><\/td>\n1<\/td>\n22<\/td>\n29<\/td>\n-7<\/td>\n-2<\/td>\n27<\/td>\n<\/tr>\n
    F<\/strong><\/td>\n0<\/td>\n33<\/td>\n29<\/td>\n4<\/td>\n3<\/td>\n32<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n

    Error values undergo adjustment with a new prediction, referred to as ‘Prediction 2’. By incorporating the previously calculated mean and ‘Prediction 2’, our aim is to approximate the actual age value. Consequently, the error or discrepancy between the actual and predicted values should diminish.<\/p>\n

    Furthermore, this iterative process continues, with each iteration utilizing the residual from the current prediction to inform the output of the next stage. The iterative nature of this process enables continuous refinement of predictions, gradually reducing the error over successive iterations.<\/p>\n

    Moreover, this iterative process persists until the maximum number of iterations is reached.<\/p>\n

    Now, let’s delve into an example to illustrate this process.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

     <\/p>\n<\/div>\n

    Example:<\/span><\/div>\n<\/div>\n<\/div>\n<\/div>\n

    We will take the same example as the AdaBoosting technique. We will apply GBM on training and testing sets of the dataset.<\/p>\n

    from sklearn.ensemble import GradientBoostingClassifier<\/p>\n

    model= GradientBoostingClassifier(learning_rate=0.01,random_state=1)<\/p>\n

    model.fit(x_train, y_train)<\/p>\n

    model.score(x_test,y_test)<\/p>\n

    Output:<\/strong><\/h4>\n

    accuracy_score on test dataset :\u00a0 0.7595555555555555<\/p>\n

    Note: In the case of regression, the steps will be the same. Just we have to replace GradientBoostingClassifier with GradientBoostingRegressor.<\/p>\n

    Light GBM Machine Learning:<\/h3>\n

    Advantages of Light GBM for Large Datasets<\/strong><\/p>\n

    When dealing with extremely large datasets, Light GBM emerges as a highly effective solution. This technique boasts superior speed when processing vast amounts of data compared to other algorithms. Unlike traditional level-wise approaches employed by certain algorithms, Light GBM utilizes a tree-based algorithm and adopts a leaf-wise approach.<\/p>\n

    Leaf-Wise Approach<\/strong><\/p>\n

    In the leaf-wise approach, Light GBM constructs trees by splitting nodes based on the leaf that results in the maximum reduction of the loss function. This strategy enhances efficiency, particularly on large datasets, as it minimizes the number of levels in each tree.<\/p>\n

    However, it’s worth noting that the leaf-wise approach may lead to overfitting, especially on smaller datasets. To mitigate this risk, Light GBM provides a parameter called ‘max_depth’, which enables control over the maximum depth of each tree.<\/p>\n

    By leveraging these advantages and carefully tuning parameters like ‘max_depth’, Light GBM offers a powerful solution for handling large datasets efficiently while minimizing the risk of overfitting.<\/p>\n

    We will apply the Light GBM train data set and the model will predict for the test dataset.<\/h4>\n

    First, install function<\/p>\n

    pip install lightgbm<\/p>\n

    Applying the algorithm to data sets<\/h4>\n

    import lightgbm as lgb<\/p>\n

    train_data=lgb.Dataset(x_train,label=y_train)<\/p>\n

    #defining parameters<\/strong><\/h4>\n

    params = {‘learning_rate’:0.001}<\/p>\n

    model= lgb.train(params, train_data, 100)<\/p>\n

    y_pred=model.predict(x_test)<\/p>\n

    for i in range(0,4500):\u00a0\u00a0\u00a0 # 4500 is number of datapoints iy_pred<\/p>\n

    if y_pred[i]>=0.5:<\/p>\n

    y_pred[i]=1<\/p>\n

    else:<\/p>\n

    y_pred[i]=0<\/p>\n

    # Accuracy Score on a test dataset<\/strong><\/h4>\n

    acc_test = accuracy_score(y_test,y_pred)<\/p>\n

    print(‘\\naccuracy_score on test dataset : ‘, acc_test)<\/p>\n

    Output:<\/strong><\/h4>\n

    0.958<\/p>\n

    \u00a0<\/strong><\/p>\n

    XGBoost in Machine Learning:<\/h3>\n

    XGBoost, short for Extreme Gradient Boosting, represents one of the most advanced implementations of the gradient boosting algorithm. Renowned for its exceptional speed and high predictive power, XGBoost offers significant advantages over other boosting techniques. Its efficiency is evident, as it is nearly 10 times faster compared to its counterparts, making it a preferred choice for large-scale datasets.<\/p>\n

    Moreover, XGBoost addresses common issues such as overfitting by incorporating regularization techniques. By mitigating overfitting, XGBoost enhances the overall accuracy of the model, ensuring robust performance even on complex datasets. Its regularization properties have earned it the nickname ‘regularized boosting’ technique.<\/p>\n

    Now, let’s explore the application of the XGBoost technique on the same dataset to witness its effectiveness firsthand.<\/p>\n

    Import that library<\/strong><\/h4>\n

    pip install xgboost<\/p>\n

    Applying XGBoost on the next data set.<\/b><\/h4>\n

    import xgboost as xgb<\/p>\n

    model=xgb.XGBClassifier(random_state=1,learning_rate=0.01)<\/p>\n

    model.fit(x_train, y_train)<\/p>\n

    model.score(x_test,y_test)<\/p>\n

    Output:<\/strong><\/h4>\n

    0.9995555555555555<\/p>\n

    CatBoost in Machine Learning:<\/h3>\n

    It is difficult to handle large amounts of labeled data in case of the categorical values. Means if data is having many categorical variables then it is difficult to process the label encoding operation. In that case, \u2018CatBoost\u2019 can handle categorical variables and does not require extensive data preprocessing like other ML models. Let\u2019s apply the technique to the same dataset.<\/p>\n

    Import library<\/strong><\/h3>\n

     <\/p>\n

    pip install catboost<\/p>\n

    Applying CatBoost<\/strong><\/h3>\n

    from catboost import CatBoostClassifier<\/p>\n

    model=CatBoostClassifier()<\/p>\n

    categorical_features_indices = np.where(df.dtypes != np.float)[0]<\/p>\n

    model.fit(x_train,y_train,cat_features=([0,1,2,3,4]),eval_set=(x_test, y_test))<\/p>\n

    model.score(x_test,y_test)<\/p>\n

    Output:<\/strong><\/h4>\n

    0.9991111111111111<\/p>\n

    We hope you understand the boosting techniques in machine learning. Get success in your career as a Data Scientist<\/a> by being a part of the Prwatech<\/a>, India’s leading Data Science training institute in Bangalore<\/a>.<\/p>\n