Bagging Technique in Machine Learning
Bagging Technique in Machine Learning
Ask a hundred people to guess how many sweets are in a jar, and the average of their guesses is often remarkably close to the truth. No single guess is reliable, some are far too high and some far too low, but the mistakes tend to cancel out. Bagging applies the same idea to machine learning. Instead of trusting one model, it trains many and combines their answers.
The name is short for bootstrap aggregating. It is designed to cure one specific weakness: variance. Some models, especially decision trees, are very sensitive to the exact data they were trained on. Change a few rows and you get a quite different tree. Bagging smooths that out. It is the technique behind random forests, which are among the most widely used models for tables of data.
This guide explains how bagging works, starting with bootstrap sampling and the reason averaging helps. Then we build a complete Python example that compares a single decision tree, a bagging model and a random forest on the same data, and use the out of bag score as a free validation check. After that come the main settings, the variants of bagging, how it differs from boosting, its strengths and limits, common mistakes and a short FAQ.
What is Bagging?
Bagging is an ensemble method, meaning it combines several models into one. Its recipe has three parts:
- Create many different training sets by drawing rows at random from the original data, with replacement.
- Fit one model, usually a decision tree, on each training set, all independently.
- Combine the predictions. For classification, the models vote or their probabilities are averaged. For regression, their predictions are averaged.
Because the models are trained independently, they can be trained in parallel, which makes bagging easy to speed up.
Bootstrap Sampling: The Heart of Bagging
Drawing with replacement means a row can be picked more than once, and some rows are not picked at all. If your data has n rows and you draw n times, roughly 63 percent of the original rows end up in the sample, and about 37 percent are left out. Each model therefore sees a slightly different version of the data, which is exactly what makes the models differ from one another.
The rows left out of a given model’s sample are called its out of bag rows. They are useful, as you will see below, because they act as a free test set.
Why does Averaging Reduce Variance?
A deep decision tree fits its training data very closely, including its noise. That makes it low in bias but high in variance. Each tree errs in its own random direction. When you average many trees that err in different directions, the random errors largely cancel, while the real pattern, which all trees share, remains. The result is a model that is more stable and usually more accurate, without being any simpler.
This works best with models that are strong but unstable, like full depth trees. It does little for models that are already stable, such as linear regression.
Bagging Versus Boosting
| Point | Bagging | Boosting |
|---|---|---|
| Main goal | Reduce variance | Reduce bias |
| Build order | Models trained independently, in parallel | Models trained one after another |
| Best base models | Strong and complex, such as deep trees | Weak and simple, such as shallow trees |
| Data for each model | Random bootstrap samples | Re weighted data or fitted to the errors |
| Overfitting risk | Low | Higher if run too long |
| Examples | Bagging classifier, random forest | AdaBoost, XGBoost, LightGBM |
Bagging in Python Step by Step
We will use the breast cancer dataset that ships with scikit learn, which has 569 patients and 30 measurements each, labelled malignant or benign. First, load the data and split it into training and test sets:
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
print(X.shape)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=0, stratify=y
)
Now the main program. It trains three models on the same split: one decision tree, a bagging model of 100 trees with the out of bag score switched on, and a random forest of 100 trees.
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=0, stratify=y
)
tree = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
bagging = BaggingClassifier(
estimator=DecisionTreeClassifier(),
n_estimators=100,
oob_score=True,
random_state=0,
).fit(X_train, y_train)
forest = RandomForestClassifier(n_estimators=100, random_state=0).fit(X_train, y_train)
print(“Single tree test accuracy:”, round(tree.score(X_test, y_test), 3))
print(“Bagging test accuracy:”, round(bagging.score(X_test, y_test), 3))
print(“Bagging out of bag score:”, round(bagging.oob_score_, 3))
print(“Random forest test accuracy:”, round(forest.score(X_test, y_test), 3))
print(“Random forest training accuracy:”, round(forest.score(X_train, y_train), 3))
When I ran it, this is what it printed:
Single tree test accuracy: 0.906
Bagging test accuracy: 0.936
Bagging out of bag score: 0.955
Random forest test accuracy: 0.953
Random forest training accuracy: 1.0
Reading the Results
- Bagging beat the single tree.The same type of model, trained 100 times on bootstrap samples, gained about three points of accuracy. I also checked this with 10 fold cross validation, where the tree averaged 0.917 and bagging 0.958.
- The out of bag score is a free check.It scored each training row using only the trees that never saw it, and gave 0.955 without using the test set. It is a good early indication of how the model will behave on new data.
- Random forest did a little better still.It scored 0.953, because it adds one more source of randomness, covered below.
- Training accuracy of 1.0 is normal here.A forest of fully grown trees memorises its training data. What matters is the test score.
On a test set of 171 patients, one patient is worth about 0.6 points, so the gap between bagging and the forest is within the noise. Do not read it as a winner.
Key Settings of BaggingClassifier
| Setting | Default | What it does |
|---|---|---|
| estimator | Decision tree | The base model trained on each sample |
| n_estimators | 10 | How many models to build. More is steadier but slower |
| max_samples | All rows | How many rows each model draws |
| max_features | 1.0 | Share of features each model sees |
| bootstrap | True | Draw rows with replacement. False draws without |
| bootstrap_features | False | Draw features with replacement |
| oob_score | False | Compute the out of bag score. Needs bootstrap on |
| n_jobs | None | Parallel jobs. Use minus one to use every core |
Predictions are combined by averaging each model’s predicted class probabilities. If the base model cannot give probabilities, the ensemble uses a simple vote.
Variants of Bagging
- Draw rows without replacement.
- Draw rows with replacement, the standard method.
- Random subspaces.Each model sees a random subset of the features.
- Random patches.Each model sees random subsets of both rows and features.
Random Forest: Bagging With One Extra Trick
A random forest is bagging with decision trees plus one addition. At every split inside every tree, only a random subset of the features is considered. This forces the trees to differ more from each other, so their errors cancel better. It is why a random forest usually matches or beats plain bagging, and why it needs little tuning.
In classic descriptions a forest predicts by majority vote. In scikit learn, it averages the predicted class probabilities of its trees and picks the class with the highest average. The two usually give the same answer.
Strengths and Limitations of Bagging
Strengths
- Cuts variance and often improves accuracy over a single model.
- Hard to overfit by adding more models.
- Models train in parallel.
- The out of bag score gives a built in validation estimate.
- Works with almost any base model.
Limitations
- Does not reduce bias. If the base model is too simple, bagging will not rescue it.
- Harder to interpret than a single tree.
- Uses more memory and prediction time, because you store and run many models.
- Gives little gain for stable models.
Applications of Bagging
- Credit risk and fraud detection. Random forests are common in finance.
- Predicting disease risk from patient measurements.
- Customer churn. Finding customers likely to leave.
- Price and demand prediction. Averaging trees for regression.
- Feature importance. Forests give a quick ranking of which inputs matter.
Common Mistakes to Avoid
- Judging by training accuracy. Fully grown trees always look perfect on their own training data.
- Using too few models. With the default of 10, results can vary. Try 100 or more.
- Bagging a stable model. Linear models gain very little.
- Switching on oob_score without bootstrap. The out of bag score needs bootstrap sampling.
- Reading a small gap as a winner. Use cross validation before choosing between methods.
- Fitting twice. Calling fit repeatedly wastes time and can hide mistakes.
Bagging shows that averaging many imperfect models beats trusting one. Run the comparison program on your own data, check the out of bag score, and remember that its real job is reducing variance. When you need to reduce bias instead, look at boosting.
