{"id":4824,"date":"2026-10-14T11:40:01","date_gmt":"2026-10-14T11:40:01","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=4824"},"modified":"2026-10-15T15:46:56","modified_gmt":"2026-10-15T15:46:56","slug":"ensemble-methods-tutorial","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/ensemble-methods-tutorial\/","title":{"rendered":"Ensemble Methods"},"content":{"rendered":"

Ensemble Methods in Machine Learning<\/b><\/strong><\/h1>\r\n

Before you buy something expensive, you probably ask more than one person. A friend may know the product, a colleague may have used it, and a review site may have tested it. No single opinion is perfect, but together they usually point you in a better direction than any one of them alone. Ensemble methods apply exactly this habit to machine learning.<\/p>\r\n

An ensemble combines the predictions of several models into a single answer. The reasoning is that different models make different mistakes. If they are combined well, the mistakes partly cancel and the group is more accurate and more stable than any individual member. The best known algorithms in applied machine learning, random forests and gradient boosted trees, are both ensembles.<\/p>\r\n

This tutorial explains every major ensemble technique in plain language. We cover the simple methods first, which are max voting, averaging and weighted averaging, with a worked example you can check by hand. Then the advanced methods: stacking, blending, bagging and boosting. After that comes a complete Python program that compares voting and stacking against the individual models on the same data, followed by advice on choosing a method, common mistakes and a short FAQ.<\/p>\r\n

What are Ensemble Methods?<\/b><\/strong><\/h2>\r\n

An ensemble method builds several models, called base models or base learners, and combines their outputs. The combination can be a vote, an average, or even another model that learns how to blend them. Ensemble methods fall into two broad families:<\/p>\r\n

    \r\n
  • <\/b>Averaging methods<\/b><\/strong>build models independently and combine them. Voting, averaging, bagging and stacking belong here. Their main benefit is lower variance.<\/li>\r\n
  • <\/b>Boosting methods<\/b><\/strong>build models one after another, each correcting the one before. Their main benefit is lower bias.<\/li>\r\n<\/ul>\r\n

    Ensemble methods are often split into simple and advanced types, which is how we will cover them.<\/p>\r\n

     <\/p>\r\n

    \"Ensemble<\/p>\r\n

    Types of Ensemble Methods<\/b><\/strong><\/h2>\r\n

    1. Simple Ensemble Methods<\/b><\/strong><\/h3>\r\n

    Max Voting<\/b><\/strong><\/h4>\r\n

    Each model votes for a class, and the class with the most votes wins. It is used mainly for classification. If three models predict spam, not spam and spam, the ensemble answers spam. This is also called hard voting. If votes are tied, the result depends on the library, so it is best to use an odd number of models.<\/p>\r\n

    Averaging<\/b><\/strong><\/h4>\r\n

    Each model produces a number, and the ensemble takes the average. For regression, the numbers are predictions. For classification, they can be predicted probabilities, which is called soft voting. Averaging smooths out the quirks of individual models.<\/p>\r\n

    Weighted Average<\/b><\/strong><\/h4>\r\n

    Some models deserve more trust than others. A weighted average gives stronger models a larger say. The following example uses predicted probabilities for three classes from three models.<\/p>\r\n

    \r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
    Soft voting example with three models<\/caption>\r\n
    Model<\/th>\r\nClass 1<\/th>\r\nClass 2<\/th>\r\nClass 3<\/th>\r\n<\/tr>\r\n<\/thead>\r\n
    Model 1<\/th>\r\n0.2<\/td>\r\n0.5<\/td>\r\n0.3<\/td>\r\n<\/tr>\r\n
    Model 2<\/th>\r\n0.6<\/td>\r\n0.3<\/td>\r\n0.1<\/td>\r\n<\/tr>\r\n
    Model 3<\/th>\r\n0.3<\/td>\r\n0.4<\/td>\r\n0.3<\/td>\r\n<\/tr>\r\n
    Equal weights average<\/th>\r\n0.37<\/td>\r\n0.40<\/td>\r\n0.23<\/td>\r\n<\/tr>\r\n
    Weights 2, 5 and 1<\/th>\r\n0.46<\/td>\r\n0.36<\/td>\r\n0.18<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/div>\r\n

     <\/p>\r\n

    With equal weights, class 2 wins with 0.40. But if Model 2 is the most trustworthy and gets a weight of 5, with weights of 2 and 1 for the others, class 1 wins with about 0.46. The data did not change, only the trust. That is why choosing weights matters, and why they should be chosen on validation data, not by hunch.<\/p>\r\n

    2. Advanced Ensemble Methods<\/b><\/strong><\/h3>\r\n

    Stacking<\/b><\/strong><\/h4>\r\n

    Stacking lets a model learn how to combine the others. The base models make predictions, and those predictions become the input features of a final model, sometimes called the meta model. The final model learns, for example, that Model 1 is reliable for some kinds of cases and Model 2 for others.<\/p>\r\n

    There is one trap. If the base models predict on the same rows they were trained on, their predictions look too good, and the final model learns to trust them too much. To avoid this, stacking uses predictions made on data the base models did not train on, produced by cross validation. Scikit learn does this for you inside StackingClassifier.<\/p>\r\n

    Blending<\/b><\/strong><\/h4>\r\n

    Blending is a simpler cousin of stacking. Instead of cross validation, you set aside a validation set, usually 10 to 20 percent of the training data. The base models are trained on the rest, they predict on the validation set, and the final model is trained on those predictions.<\/p>\r\n

    \r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
    Stacking vs blending<\/caption>\r\n
    Point<\/th>\r\nStacking<\/th>\r\nBlending<\/th>\r\n<\/tr>\r\n<\/thead>\r\n
    Final model trained on<\/th>\r\nCross validated predictions for every training row<\/td>\r\nPredictions on one held out validation set<\/td>\r\n<\/tr>\r\n
    Data used<\/th>\r\nAll the training data<\/td>\r\nLess, because some is held out<\/td>\r\n<\/tr>\r\n
    Complexity<\/th>\r\nMore, and slower<\/td>\r\nSimpler and faster<\/td>\r\n<\/tr>\r\n
    Leakage risk<\/th>\r\nHandled by cross validation<\/td>\r\nLow, if the split is kept separate<\/td>\r\n<\/tr>\r\n
    In scikit learn<\/th>\r\nStackingClassifier and StackingRegressor<\/td>\r\nNo ready made class, write it by hand<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/div>\r\n

     <\/p>\r\n

    Bagging<\/b><\/strong><\/h4>\r\n

    Bagging, short for bootstrap aggregating, trains many models in parallel, each on a random sample of the data drawn with replacement, and averages them. It reduces variance and works best with strong, unstable models such as deep decision trees. Random forest is bagging plus a random choice of features at each split. Our full guide to bagging covers it in detail.<\/p>\r\n

    Boosting<\/b><\/strong><\/h4>\r\n

    Boosting trains simple models in sequence. Each new model concentrates on the mistakes of the ones before it, and the final prediction combines them all. It reduces bias and often gives the highest accuracy on tables of data. AdaBoost, gradient boosting, XGBoost, LightGBM and CatBoost are all boosting methods. Our full guide to boosting covers them.<\/p>\r\n

     <\/p>\r\n

     <\/p>\r\n