☰

Decision Tree Introduction with Examples

Decision Tree in Machine Learning: Introduction With Examples

Think about how a doctor narrows down a diagnosis. Is there a fever? If yes, is there a cough? If yes, is the oxygen level low? Each answer rules out some possibilities and points toward others, until a conclusion is left. A decision tree works the same way. It asks a chain of simple questions about your data, and each answer sends the case down one branch until it reaches a final answer.

Decision trees are among the easiest machine learning models to understand, because you can draw them and read them like a flowchart. They handle both classification, where the answer is a category, and regression, where the answer is a number. They need little data preparation, and they are the building block of random forests and boosting, two of the most successful methods for tables of data.

This tutorial explains how a decision tree works and how it decides which question to ask first. We cover the key terms, then the three measures that choose each split: entropy, information gain and Gini impurity. A worked example with every number calculated by running code makes it concrete. Then comes a complete Python program, run and checked, that shows a tree you can read, how depth causes overfitting, and how pruning fixes it. We finish with strengths, limits, common mistakes and an FAQ.

What is a Decision Tree?

A decision tree is a supervised learning model that makes predictions by following a set of if then rules arranged as a tree. Each internal node tests one feature. Each branch is an outcome of that test. Each leaf at the end gives the final prediction, which is a class for classification and an average value for regression.

Common Terms Used With Decision Trees

Decision tree terms
Term Meaning
Root node The top node, where the first split is made. It holds all the data
Splitting Dividing a node into two or more sub nodes using a question
Decision node A node that splits further into more nodes
Leaf or terminal node A node that does not split. It gives the final answer
Branch or sub tree A section of the tree
Parent and child node A node that splits is the parent of the nodes it creates
Pruning Removing branches to make the tree simpler and reduce overfitting

Decision tree Introduction with examples - Different terms

How is a Decision Tree Created?

A tree is built from the top down. At each node, the algorithm tries the possible questions, measures how much each one improves the purity of the groups, and chooses the best. It then repeats on each new group. It stops when a group is pure, meaning it holds only one class, or when no question helps, or when a limit such as maximum depth is reached.

The key question is how to measure a good split. Three measures are used.

Entropy

Entropy measures uncertainty, or how mixed a group is. A group that is all one class has entropy 0, since there is no uncertainty. A group split evenly between two classes has entropy 1, the highest for two classes. Higher entropy means more uncertainty and a more mixed group. The formula is the negative sum, over every class, of the class proportion times the base 2 logarithm of that proportion.

For a group of 10 items with 4 of class a and 6 of class b, the proportions are 0.4 and 0.6. The entropy comes to 0.971, which is quite mixed.

Information Gain

Information gain is the drop in entropy after a split. You take the entropy of the parent group, subtract the weighted average entropy of the child groups, and what remains is the gain. The weights are the share of rows that goes to each child. The feature with the highest information gain is chosen for the split.

Information Gain Formula

The tree stops splitting a node when it is pure, or when no feature gives a gain above zero.

Decision tree Introduction with examples - Information Gain

Gini Impurity

Gini impurity measures the chance that a randomly chosen item would be wrongly labelled if you labelled it at random, using the class proportions in the group. The formula is one minus the sum of the squared class proportions. A pure group scores 0. For two classes, an even mix scores 0.5, the worst case.

For the same group of 4 and 6, the proportions are 0.4 and 0.6. The Gini is 1 minus (0.16 plus 0.36), which is 0.48. Scikit learn uses Gini by default. It is a little faster to compute, and in practice it picks very similar splits to entropy.

Entropy and Gini from a pure group to a mixed group
Measure Pure group Even two class mix Group of 4 and 6
Entropy 0 1 0.971
Gini impurity 0 0.5 0.48

 

Decision Tree Example: Will We Play Cricket?

Here is a worked example with a complete table. Over 10 days, we noted the weather, humidity and wind, and whether the match was played. The values are made up to show the method.

Decision tree Introduction with examples

Cricket play data for 10 days
Day Weather Humidity Wind Play cricket
1 Sunny High Weak No
2 Sunny High Strong No
3 Cloudy High Weak Yes
4 Rainy High Weak Yes
5 Rainy Normal Weak Yes
6 Rainy Normal Strong No
7 Cloudy Normal Strong Yes
8 Sunny High Weak No
9 Sunny Normal Weak Yes
10 Rainy Normal Weak Yes

 

Step 1: Entropy of the Whole Table

The match was played on 6 days and not played on 4. That is the same 4 and 6 split from before, so the entropy of the target is 0.971.

Entropy Example

Where TT= Total number of targets=14

Decision tree Introduction with examples - Entropy Example

Entrop Example

Step 2: Information Gain of Each Feature

For each feature, we split the table by its values, calculate the entropy of each part, weight it by the share of rows, and subtract the total from 0.971. Here is the Weather feature:

Entropy of each weather group
Weather Days Played Entropy of the group
Cloudy 2 2 0 (pure, always played)
Rainy 4 3 0.811
Sunny 4 1 0.811

 

The weighted entropy is 0.2 times 0 plus 0.4 times 0.811 plus 0.4 times 0.811, which is 0.649. The gain is 0.971 minus 0.649, or 0.322. Note that the number 0.649 is the weighted entropy, which the original page loosely calls information.

Decision Tree example

Here are the gains for all three features. I calculated these by running the code below.

Information gain for each feature
Feature Weighted entropy Information gain
Weather 0.649 0.322
Humidity 0.846 0.125
Wind 0.880 0.091

Decision tree Entropy

python
import numpy as np
import pandas as pd

df = pd.DataFrame({
    "Weather":  ["Sunny", "Sunny", "Cloudy", "Rainy", "Rainy", "Rainy", "Cloudy", "Sunny", "Sunny", "Rainy"],
    "Humidity": ["High", "High", "High", "High", "Normal", "Normal", "Normal", "High", "Normal", "Normal"],
    "Wind":     ["Weak", "Strong", "Weak", "Weak", "Weak", "Strong", "Strong", "Weak", "Weak", "Weak"],
    "Play":     ["No", "No", "Yes", "Yes", "Yes", "No", "Yes", "No", "Yes", "Yes"],
})

def entropy(labels):
    p = labels.value_counts(normalize=True)
    return -(p * np.log2(p)).sum()

def gini(labels):
    p = labels.value_counts(normalize=True)
    return 1 - (p ** 2).sum()

print("Play counts:", df["Play"].value_counts().to_dict())
print("Entropy of Play:", round(entropy(df["Play"]), 3))
print("Gini of Play:", round(gini(df["Play"]), 3))

for col in ["Weather", "Humidity", "Wind"]:
    weighted = 0
    for value, group in df.groupby(col):
        w = len(group) / len(df)
        e = entropy(group["Play"])
        weighted += w * e
        print(col, value, "rows", len(group), "yes", (group["Play"] == "Yes").sum(), "entropy", round(e, 3) + 0.0)
    print(col, "weighted entropy", round(weighted, 3), "information gain", round(entropy(df["Play"]) - weighted, 3))
output
Play counts: {'Yes': 6, 'No': 4}
Entropy of Play: 0.971
Gini of Play: 0.48
Weather Cloudy rows 2 yes 2 entropy 0.0
Weather Rainy rows 4 yes 3 entropy 0.811
Weather Sunny rows 4 yes 1 entropy 0.811
Weather weighted entropy 0.649 information gain 0.322
Humidity High rows 5 yes 2 entropy 0.971
Humidity Normal rows 5 yes 4 entropy 0.722
Humidity weighted entropy 0.846 information gain 0.125
Wind Strong rows 3 yes 1 entropy 0.918
Wind Weak rows 7 yes 5 entropy 0.863
Wind weighted entropy 0.88 information gain 0.091

When I ran it, the key lines were:

output
Entropy of Play: 0.971
Gini of Play: 0.48
Weather weighted entropy 0.649 information gain 0.322
Humidity weighted entropy 0.846 information gain 0.125
Wind weighted entropy 0.88 information gain 0.091

Step 3: Choose the Root and Repeat

Weather has the highest gain at 0.322, so it becomes the root node. The Cloudy branch is already pure, since it always led to play, so it becomes a leaf that says yes. The Sunny and Rainy branches still hold a mix, so the algorithm repeats the process inside each one with the remaining features, Humidity and Wind, until the groups are pure or no gain is left. The finished tree looks like this.

Decision Tree types

With only 10 rows, a deep tree would memorise the table and learn nothing general. This is a teaching example, and real trees need far more data.

Types of Decision Trees

  • Classification treespredict a category, such as play or no play. Leaves hold a class, and splits are chosen with Gini or entropy.
  • Regression treespredict a number, such as a house price. Leaves hold the average value of the rows in them, and splits are chosen to reduce the squared error.

CART, short for classification and regression trees, is the name of the overall method. It covers both types and always splits into two branches. It is the version scikit learn uses.

Overfitting and Pruning

A tree left to grow without limit keeps splitting until every leaf is pure. It then fits its training data perfectly, noise included, and fails on new data. This is overfitting, and it is the main weakness of decision trees. Pruning fixes it by cutting back branches that add little. There are two ways:

  • Pre pruning.Stop early, using limits such as maximum depth or a minimum number of rows in a leaf.
  • Post pruning.Grow the full tree, then cut back the weakest branches. Scikit learn does this with the ccp_alpha setting. A larger value prunes more.

 

Decision Tree in Python With scikit learn

This program has three parts. First it trains a small tree on the Iris flower dataset and prints it as readable rules. Then it uses the breast cancer dataset, with 569 patients and 30 measurements each, to show how depth and pruning change the result. Last it compares Gini with entropy. Both datasets come with scikit learn, so there is no file to load.

python
from sklearn.datasets import load_breast_cancer, load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier, export_text

# Part 1: a small tree you can read, on the Iris data
iris = load_iris()
Xi_train, Xi_test, yi_train, yi_test = train_test_split(
    iris.data, iris.target, test_size=0.3, random_state=0, stratify=iris.target
)
small = DecisionTreeClassifier(max_depth=2, random_state=0).fit(Xi_train, yi_train)
print(export_text(small, feature_names=list(iris.feature_names)))
print("Iris test accuracy:", round(small.score(Xi_test, yi_test), 3))

# Part 2: depth, overfitting and pruning on the breast cancer data
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.3, random_state=42, stratify=data.target
)
print("depth  train  test")
for depth in [1, 2, 3, 5, 8, None]:
    m = DecisionTreeClassifier(max_depth=depth, random_state=42).fit(X_train, y_train)
    print(depth, round(m.score(X_train, y_train), 3), round(m.score(X_test, y_test), 3), "leaves:", m.get_n_leaves())

print("ccp_alpha  leaves  train  test")
for alpha in [0.0, 0.005, 0.01, 0.02]:
    m = DecisionTreeClassifier(ccp_alpha=alpha, random_state=42).fit(X_train, y_train)
    print(alpha, m.get_n_leaves(), round(m.score(X_train, y_train), 3), round(m.score(X_test, y_test), 3))

# Part 3: Gini against entropy
for criterion in ["gini", "entropy"]:
    m = DecisionTreeClassifier(criterion=criterion, max_depth=3, random_state=42).fit(X_train, y_train)
    print(criterion, round(m.score(X_test, y_test), 3))
output
|--- petal width (cm) <= 0.80
|   |--- class: 0
|--- petal width (cm) >  0.80
|   |--- petal width (cm) <= 1.75
|   |   |--- class: 1
|   |--- petal width (cm) >  1.75
|   |   |--- class: 2

Iris test accuracy: 0.956
depth  train  test
1 0.927 0.912 leaves: 2
2 0.965 0.918 leaves: 4
3 0.98 0.924 leaves: 7
5 0.995 0.93 leaves: 14
8 1.0 0.918 leaves: 16
None 1.0 0.918 leaves: 16
ccp_alpha  leaves  train  test
0.0 16 1.0 0.918
0.005 7 0.985 0.93
0.01 5 0.977 0.924
0.02 3 0.957 0.918
gini 0.924
entropy 0.947

Part 1: A Tree You Can Read

output
|--- petal width (cm) <= 0.80
|   |--- class: 0
|--- petal width (cm) >  0.80
|   |--- petal width (cm) <= 1.75
|   |   |--- class: 1
|   |--- petal width (cm) >  1.75
|   |   |--- class: 2

Iris test accuracy: 0.956

Two questions about petal width are enough to separate three species. Anyone can follow these rules without knowing any statistics, which is the great strength of a tree. The test set has 45 flowers, so 0.956 means 43 correct.

Part 2: Depth and Overfitting

output
depth  train  test
1      0.927  0.912  leaves: 2
2      0.965  0.918  leaves: 4
3      0.98   0.924  leaves: 7
5      0.995  0.93   leaves: 14
8      1.0    0.918  leaves: 16
None   1.0    0.918  leaves: 16
  • Training accuracy keeps climbing. It reaches 1.0 at depth 8, which means the tree has memorised the training set.
  • Test accuracy stops improving. It peaks at 0.93 for depth 5, then falls back to 0.918.
  • Mind the small differences. The test set has 171 patients, and one patient is worth about 0.6 points. The gap between 0.918 and 0.93 is two patients, so treat it as a rough sign, not a precise answer.
  • The tree stopped growing at 16 leaves. Depth 8 and no limit gave the same tree, because every leaf was already pure.

Part 3: Pruning

output
ccp_alpha  leaves  train  test
0.0        16      1.0    0.918
0.005      7       0.985  0.93
0.01       5       0.977  0.924
0.02       3       0.957  0.918

A small amount of pruning, with ccp_alpha at 0.005, cut the tree from 16 leaves to 7 and raised test accuracy from 0.918 to 0.93. That is a simpler tree that does the same job or a little better. Prune too hard, as at 0.02, and the tree becomes too simple and the score drops.

Part 4: Gini Against Entropy

output
gini 0.924
entropy 0.947

On this data and this split, entropy scored 0.947 and Gini 0.924, a difference of about 4 patients out of 171. That does not prove one is better. In practice the two usually pick very similar trees, so do not spend time on this choice before tuning depth and pruning.

Advantages and Disadvantages of Decision Trees

Advantages

  • Easy to understand and explain, and you can draw them.
  • Handle both classification and regression.
  • Need little data preparation. They do not need scaling.
  • Handle numbers and categories, and capture curved and interacting patterns.
  • Fairly robust to outliers, because splits depend on order, not on how extreme a value is.

Disadvantages

  • Overfit easily if left to grow, which is the biggest weakness.
  • A small change in the data can produce a very different tree.
  • Predictions are steps, not smooth curves, which is a limit for regression.
  • Biased toward features with many distinct values.
  • Usually less accurate than random forests or boosting, which combine many trees.

Where Decision Trees Are Used

  • Approving or rejecting loan applications with rules that can be explained.
  • Triage and diagnosis support.
  • Deciding which customers to target.
  • Spotting faults from machine readings.
  • Inside other models. Random forests and boosting are built from many trees.

Common Mistakes to Avoid

  • Training and scoring on the same data. Always score on a held out test set.
  • Leaving the depth unlimited. Set max_depth or use pruning.
  • Trusting one tree.cCheck with cross validation or move to a random forest.
  • Reading too much into a tiny example.cA tree trained on 10 rows teaches the method, not a real result.
  • Mixing up entropy and information gain.cEntropy is the uncertainty. Gain is how much a split reduces it.
  • Using a regression tree for a category or the reverse.cMatch the tree type to the target.

A decision tree turns data into a set of questions that anyone can follow. Learn how entropy, information gain and Gini choose each split, limit the depth or prune to avoid overfitting, and always judge the tree on data it has not seen. Once you are comfortable with one tree, move on to random forest and boosting, which use many trees to get higher accuracy.

Popular Tags:

Decision tree Example Decision Tree in Machine Learning Decision Tree Introduction Decision Tree Introduction with Examples