{"id":4617,"date":"2026-10-18T10:02:44","date_gmt":"2026-10-18T10:02:44","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=4617"},"modified":"2026-10-19T14:07:34","modified_gmt":"2026-10-19T14:07:34","slug":"decision-tree-introduction","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/","title":{"rendered":"Decision Tree Introduction with Examples"},"content":{"rendered":"<h1><strong><b>Decision Tree in Machine Learning: Introduction With Examples<\/b><\/strong><\/h1>\n<p>Think about how a doctor narrows down a diagnosis. Is there a fever? If yes, is there a cough? If yes, is the oxygen level low? Each answer rules out some possibilities and points toward others, until a conclusion is left. A decision tree works the same way. It asks a chain of simple questions about your data, and each answer sends the case down one branch until it reaches a final answer.<\/p>\n<p>Decision trees are among the easiest machine learning models to understand, because you can draw them and read them like a flowchart. They handle both classification, where the answer is a category, and regression, where the answer is a number. They need little data preparation, and they are the building block of random forests and boosting, two of the most successful methods for tables of data.<\/p>\n<p>This tutorial explains how a decision tree works and how it decides which question to ask first. We cover the key terms, then the three measures that choose each split: entropy, information gain and Gini impurity. A worked example with every number calculated by running code makes it concrete. Then comes a complete Python program, run and checked, that shows a tree you can read, how depth causes overfitting, and how pruning fixes it. We finish with strengths, limits, common mistakes and an FAQ.<\/p>\n<h2><strong><b>What is a Decision Tree?<\/b><\/strong><\/h2>\n<p>A decision tree is a supervised learning model that makes predictions by following a set of if then rules arranged as a tree. Each internal node tests one feature. Each branch is an outcome of that test. Each leaf at the end gives the final prediction, which is a class for classification and an average value for regression.<\/p>\n<h3><strong><b>Common Terms Used With Decision Trees<\/b><\/strong><\/h3>\n<div style=\"border: 1px solid #d9e0ea; border-radius: 8px; overflow-x: auto; margin: 20px 0; -webkit-overflow-scrolling: touch;\">\n<table style=\"width: 100%; min-width: 660px; border-collapse: collapse; font-family: Arial,Helvetica,sans-serif; font-size: 16px; line-height: 1.6; color: #1f2937;\">\n<caption style=\"caption-side: top; text-align: left; padding: 14px 16px; font-weight: bold; font-size: 17px; color: #0f2a4a;\">Decision tree terms<\/caption>\n<thead>\n<tr>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38; width: 230px;\" scope=\"col\">Term<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38;\" scope=\"col\">Meaning<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">Root node<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">The top node, where the first split is made. It holds all the data<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">Splitting<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Dividing a node into two or more sub nodes using a question<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">Decision node<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">A node that splits further into more nodes<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">Leaf or terminal node<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">A node that does not split. It gives the final answer<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">Branch or sub tree<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">A section of the tree<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">Parent and child node<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">A node that splits is the parent of the nodes it creates<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; color: #0f2a4a;\" scope=\"row\">Pruning<\/th>\n<td style=\"padding: 12px 16px;\">Removing branches to make the tree simpler and reduce overfitting<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4731\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre2.png\" alt=\"Decision tree Introduction with examples - Different terms\" width=\"850\" height=\"606\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre2.png 864w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre2-300x214.png 300w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre2-768x548.png 768w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<div class=\"flex-1 overflow-hidden\">\n<div class=\"react-scroll-to-bottom--css-koqot-79elbk h-full\">\n<div class=\"react-scroll-to-bottom--css-koqot-1n7m0yu\">\n<div class=\"flex flex-col text-sm pb-9\">\n<div class=\"w-full text-token-text-primary\" data-testid=\"conversation-turn-7\">\n<div class=\"px-4 py-2 justify-center text-base md:gap-6 m-auto\">\n<div class=\"flex flex-1 text-base mx-auto gap-3 md:px-5 lg:px-1 xl:px-5 md:max-w-3xl lg:max-w-[40rem] xl:max-w-[48rem] group final-completion\">\n<div class=\"relative flex w-full flex-col agent-turn\">\n<div class=\"flex-col gap-1 md:gap-3\">\n<div class=\"flex flex-grow flex-col max-w-full\">\n<div class=\"min-h-[20px] text-message flex flex-col items-start gap-3 whitespace-pre-wrap break-words [.text-message+&amp;]:mt-5 overflow-x-auto\" data-message-author-role=\"assistant\" data-message-id=\"364afa75-434a-4cae-8d96-e482ca0f572f\">\n<div class=\"markdown prose w-full break-words dark:prose-invert light\">\n<h2><strong><b>How is a Decision Tree Created?<\/b><\/strong><\/h2>\n<p>A tree is built from the top down. At each node, the algorithm tries the possible questions, measures how much each one improves the purity of the groups, and chooses the best. It then repeats on each new group. It stops when a group is pure, meaning it holds only one class, or when no question helps, or when a limit such as maximum depth is reached.<\/p>\n<p>The key question is how to measure a good split. Three measures are used.<\/p>\n<h3><strong><b>Entropy<\/b><\/strong><\/h3>\n<p>Entropy measures uncertainty, or how mixed a group is. A group that is all one class has entropy 0, since there is no uncertainty. A group split evenly between two classes has entropy 1, the highest for two classes. Higher entropy means more uncertainty and a more mixed group. The formula is the negative sum, over every class, of the class proportion times the base 2 logarithm of that proportion.<\/p>\n<p>For a group of 10 items with 4 of class a and 6 of class b, the proportions are 0.4 and 0.6. The entropy comes to 0.971, which is quite mixed.<\/p>\n<h3><strong><b>Information Gain<\/b><\/strong><\/h3>\n<p>Information gain is the drop in entropy after a split. You take the entropy of the parent group, subtract the weighted average entropy of the child groups, and what remains is the gain. The weights are the share of rows that goes to each child. The feature with the highest information gain is chosen for the split.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4884\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/information-gain.png\" alt=\"Information Gain Formula\" width=\"850\" height=\"96\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/information-gain.png 442w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/information-gain-300x34.png 300w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<p>The tree stops splitting a node when it is pure, or when no feature gives a gain above zero.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4885\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Information-Gain-Example.png\" alt=\"Decision tree Introduction with examples - Information Gain \" width=\"850\" height=\"254\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Information-Gain-Example.png 415w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Information-Gain-Example-300x90.png 300w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<h3><strong><b>Gini Impurity<\/b><\/strong><\/h3>\n<p>Gini impurity measures the chance that a randomly chosen item would be wrongly labelled if you labelled it at random, using the class proportions in the group. The formula is one minus the sum of the squared class proportions. A pure group scores 0. For two classes, an even mix scores 0.5, the worst case.<\/p>\n<p>For the same group of 4 and 6, the proportions are 0.4 and 0.6. The Gini is 1 minus (0.16 plus 0.36), which is 0.48. Scikit learn uses Gini by default. It is a little faster to compute, and in practice it picks very similar splits to entropy.<\/p>\n<div style=\"border: 1px solid #d9e0ea; border-radius: 8px; overflow-x: auto; margin: 20px 0; -webkit-overflow-scrolling: touch;\">\n<table style=\"width: 100%; min-width: 700px; border-collapse: collapse; font-family: Arial,Helvetica,sans-serif; font-size: 16px; line-height: 1.6; color: #1f2937;\">\n<caption style=\"caption-side: top; text-align: left; padding: 14px 16px; font-weight: bold; font-size: 17px; color: #0f2a4a;\">Entropy and Gini from a pure group to a mixed group<\/caption>\n<thead>\n<tr>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38; width: 180px;\" scope=\"col\">Measure<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38;\" scope=\"col\">Pure group<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38;\" scope=\"col\">Even two class mix<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38;\" scope=\"col\">Group of 4 and 6<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Entropy<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">0<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">1<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">0.971<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Gini impurity<\/th>\n<td style=\"padding: 12px 16px;\">0<\/td>\n<td style=\"padding: 12px 16px;\">0.5<\/td>\n<td style=\"padding: 12px 16px;\">0.48<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>&nbsp;<\/p>\n<h2><strong><b>Decision Tree Example: Will We Play Cricket?<\/b><\/strong><\/h2>\n<p>Here is a worked example with a complete table. Over 10 days, we noted the weather, humidity and wind, and whether the match was played. The values are made up to show the method.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4886\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Decision-Tree-Example.png\" alt=\"Decision tree Introduction with examples\" width=\"850\" height=\"260\" \/><\/p>\n<div style=\"border: 1px solid #d9e0ea; border-radius: 8px; overflow-x: auto; margin: 20px 0; -webkit-overflow-scrolling: touch;\">\n<table style=\"width: 100%; min-width: 640px; border-collapse: collapse; font-family: Arial,Helvetica,sans-serif; font-size: 16px; line-height: 1.6; color: #1f2937;\">\n<caption style=\"caption-side: top; text-align: left; padding: 14px 16px; font-weight: bold; font-size: 17px; color: #0f2a4a;\">Cricket play data for 10 days<\/caption>\n<thead>\n<tr>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38; width: 80px;\" scope=\"col\">Day<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38;\" scope=\"col\">Weather<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38;\" scope=\"col\">Humidity<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38;\" scope=\"col\">Wind<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38;\" scope=\"col\">Play cricket<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">1<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Sunny<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">High<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Weak<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">No<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">2<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Sunny<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">High<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Strong<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">No<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">3<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Cloudy<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">High<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Weak<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Yes<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">4<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Rainy<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">High<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Weak<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Yes<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">5<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Rainy<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Normal<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Weak<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Yes<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">6<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Rainy<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Normal<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Strong<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">No<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">7<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Cloudy<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Normal<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Strong<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Yes<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">8<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Sunny<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">High<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Weak<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">No<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">9<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Sunny<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Normal<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Weak<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Yes<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; color: #0f2a4a;\" scope=\"row\">10<\/th>\n<td style=\"padding: 12px 16px;\">Rainy<\/td>\n<td style=\"padding: 12px 16px;\">Normal<\/td>\n<td style=\"padding: 12px 16px;\">Weak<\/td>\n<td style=\"padding: 12px 16px;\">Yes<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>&nbsp;<\/p>\n<h3><strong><b>Step 1: Entropy of the Whole Table<\/b><\/strong><\/h3>\n<p>The match was played on 6 days and not played on 4. That is the same 4 and 6 split from before, so the entropy of the target is 0.971.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4887\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Entropy-1.png\" alt=\"Entropy Example\" width=\"850\" height=\"161\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Entropy-1.png 412w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Entropy-1-300x57.png 300w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<p>Where TT= Total number of targets=14<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4888\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Entropy-Example.png\" alt=\"Decision tree Introduction with examples - Entropy Example\" width=\"850\" height=\"282\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Entropy-Example.png 356w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Entropy-Example-300x99.png 300w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4889\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Entrop-Example-in-Decision-Tree.png\" alt=\"Entrop Example \" width=\"850\" height=\"532\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Entrop-Example-in-Decision-Tree.png 355w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Entrop-Example-in-Decision-Tree-300x188.png 300w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<h3><strong><b>Step 2: Information Gain of Each Feature<\/b><\/strong><\/h3>\n<p>For each feature, we split the table by its values, calculate the entropy of each part, weight it by the share of rows, and subtract the total from 0.971. Here is the Weather feature:<\/p>\n<div style=\"border: 1px solid #d9e0ea; border-radius: 8px; overflow-x: auto; margin: 20px 0; -webkit-overflow-scrolling: touch;\">\n<table style=\"width: 100%; min-width: 660px; border-collapse: collapse; font-family: Arial,Helvetica,sans-serif; font-size: 16px; line-height: 1.6; color: #1f2937;\">\n<caption style=\"caption-side: top; text-align: left; padding: 14px 16px; font-weight: bold; font-size: 17px; color: #0f2a4a;\">Entropy of each weather group<\/caption>\n<thead>\n<tr>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38; width: 170px;\" scope=\"col\">Weather<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38; width: 120px;\" scope=\"col\">Days<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38; width: 120px;\" scope=\"col\">Played<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38;\" scope=\"col\">Entropy of the group<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Cloudy<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">2<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">2<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">0 (pure, always played)<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Rainy<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">4<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">3<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">0.811<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Sunny<\/th>\n<td style=\"padding: 12px 16px;\">4<\/td>\n<td style=\"padding: 12px 16px;\">1<\/td>\n<td style=\"padding: 12px 16px;\">0.811<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>&nbsp;<\/p>\n<p>The weighted entropy is 0.2 times 0 plus 0.4 times 0.811 plus 0.4 times 0.811, which is 0.649. The gain is 0.971 minus 0.649, or 0.322. Note that the number 0.649 is the weighted entropy, which the original page loosely calls information.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4890\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Decision-Tree-Entropy-Example.png\" alt=\"Decision Tree example\" width=\"850\" height=\"221\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Decision-Tree-Entropy-Example.png 534w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Decision-Tree-Entropy-Example-300x78.png 300w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<p>Here are the gains for all three features. I calculated these by running the code below.<\/p>\n<div style=\"border: 1px solid #d9e0ea; border-radius: 8px; overflow-x: auto; margin: 20px 0; -webkit-overflow-scrolling: touch;\">\n<table style=\"width: 100%; min-width: 640px; border-collapse: collapse; font-family: Arial,Helvetica,sans-serif; font-size: 16px; line-height: 1.6; color: #1f2937;\">\n<caption style=\"caption-side: top; text-align: left; padding: 14px 16px; font-weight: bold; font-size: 17px; color: #0f2a4a;\">Information gain for each feature<\/caption>\n<thead>\n<tr>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38; width: 200px;\" scope=\"col\">Feature<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38;\" scope=\"col\">Weighted entropy<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38;\" scope=\"col\">Information gain<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Weather<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">0.649<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">0.322<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Humidity<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">0.846<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">0.125<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Wind<\/th>\n<td style=\"padding: 12px 16px;\">0.880<\/td>\n<td style=\"padding: 12px 16px;\">0.091<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4891\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Decision-Tree-Entropy-Example-1.png\" alt=\"Decision tree Entropy\" width=\"850\" height=\"668\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Decision-Tree-Entropy-Example-1.png 533w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/Decision-Tree-Entropy-Example-1-300x236.png 300w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<div style=\"margin: 20px 0; border: 1px solid #d9e0ea; border-radius: 8px; overflow: hidden;\">\n<div style=\"background: #0f2a4a; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">python<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #f5f8fc; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">import numpy as np\r\nimport pandas as pd\r\n\r\ndf = pd.DataFrame({\r\n    \"Weather\":  [\"Sunny\", \"Sunny\", \"Cloudy\", \"Rainy\", \"Rainy\", \"Rainy\", \"Cloudy\", \"Sunny\", \"Sunny\", \"Rainy\"],\r\n    \"Humidity\": [\"High\", \"High\", \"High\", \"High\", \"Normal\", \"Normal\", \"Normal\", \"High\", \"Normal\", \"Normal\"],\r\n    \"Wind\":     [\"Weak\", \"Strong\", \"Weak\", \"Weak\", \"Weak\", \"Strong\", \"Strong\", \"Weak\", \"Weak\", \"Weak\"],\r\n    \"Play\":     [\"No\", \"No\", \"Yes\", \"Yes\", \"Yes\", \"No\", \"Yes\", \"No\", \"Yes\", \"Yes\"],\r\n})\r\n\r\ndef entropy(labels):\r\n    p = labels.value_counts(normalize=True)\r\n    return -(p * np.log2(p)).sum()\r\n\r\ndef gini(labels):\r\n    p = labels.value_counts(normalize=True)\r\n    return 1 - (p ** 2).sum()\r\n\r\nprint(\"Play counts:\", df[\"Play\"].value_counts().to_dict())\r\nprint(\"Entropy of Play:\", round(entropy(df[\"Play\"]), 3))\r\nprint(\"Gini of Play:\", round(gini(df[\"Play\"]), 3))\r\n\r\nfor col in [\"Weather\", \"Humidity\", \"Wind\"]:\r\n    weighted = 0\r\n    for value, group in df.groupby(col):\r\n        w = len(group) \/ len(df)\r\n        e = entropy(group[\"Play\"])\r\n        weighted += w * e\r\n        print(col, value, \"rows\", len(group), \"yes\", (group[\"Play\"] == \"Yes\").sum(), \"entropy\", round(e, 3) + 0.0)\r\n    print(col, \"weighted entropy\", round(weighted, 3), \"information gain\", round(entropy(df[\"Play\"]) - weighted, 3))<\/pre>\n<div style=\"background: #1f2937; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">output<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #ffffff; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">Play counts: {'Yes': 6, 'No': 4}\r\nEntropy of Play: 0.971\r\nGini of Play: 0.48\r\nWeather Cloudy rows 2 yes 2 entropy 0.0\r\nWeather Rainy rows 4 yes 3 entropy 0.811\r\nWeather Sunny rows 4 yes 1 entropy 0.811\r\nWeather weighted entropy 0.649 information gain 0.322\r\nHumidity High rows 5 yes 2 entropy 0.971\r\nHumidity Normal rows 5 yes 4 entropy 0.722\r\nHumidity weighted entropy 0.846 information gain 0.125\r\nWind Strong rows 3 yes 1 entropy 0.918\r\nWind Weak rows 7 yes 5 entropy 0.863\r\nWind weighted entropy 0.88 information gain 0.091<\/pre>\n<\/div>\n<p>When I ran it, the key lines were:<\/p>\n<div style=\"margin: 20px 0; border: 1px solid #d9e0ea; border-radius: 8px; overflow: hidden;\">\n<div style=\"background: #1f2937; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">output<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #ffffff; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">Entropy of Play: 0.971\r\nGini of Play: 0.48\r\nWeather weighted entropy 0.649 information gain 0.322\r\nHumidity weighted entropy 0.846 information gain 0.125\r\nWind weighted entropy 0.88 information gain 0.091<\/pre>\n<\/div>\n<h3><strong><b>Step 3: Choose the Root and Repeat<\/b><\/strong><\/h3>\n<p>Weather has the highest gain at 0.322, so it becomes the root node. The Cloudy branch is already pure, since it always led to play, so it becomes a leaf that says yes. The Sunny and Rainy branches still hold a mix, so the algorithm repeats the process inside each one with the remaining features, Humidity and Wind, until the groups are pure or no gain is left. The finished tree looks like this.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4892\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre1-1-1024x565.png\" alt=\"Decision Tree types\" width=\"850\" height=\"469\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre1-1-1024x565.png 1024w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre1-1-300x166.png 300w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre1-1-768x424.png 768w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre1-1.png 1060w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<p>With only 10 rows, a deep tree would memorise the table and learn nothing general. This is a teaching example, and real trees need far more data.<\/p>\n<h2><strong><b>Types of Decision Trees<\/b><\/strong><\/h2>\n<ul>\n<li><b><\/b><strong><b>Classification trees<\/b><\/strong>predict a category, such as play or no play. Leaves hold a class, and splits are chosen with Gini or entropy.<\/li>\n<li><b><\/b><strong><b>Regression trees<\/b><\/strong>predict a number, such as a house price. Leaves hold the average value of the rows in them, and splits are chosen to reduce the squared error.<\/li>\n<\/ul>\n<p>CART, short for classification and regression trees, is the name of the overall method. It covers both types and always splits into two branches. It is the version scikit learn uses.<\/p>\n<h2><strong><b>Overfitting and Pruning<\/b><\/strong><\/h2>\n<p>A tree left to grow without limit keeps splitting until every leaf is pure. It then fits its training data perfectly, noise included, and fails on new data. This is overfitting, and it is the main weakness of decision trees. Pruning fixes it by cutting back branches that add little. There are two ways:<\/p>\n<ul>\n<li><b><\/b><strong><b>Pre pruning.<\/b><\/strong>Stop early, using limits such as maximum depth or a minimum number of rows in a leaf.<\/li>\n<li><b><\/b><strong><b>Post pruning.<\/b><\/strong>Grow the full tree, then cut back the weakest branches. Scikit learn does this with the ccp_alpha setting. A larger value prunes more.<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<h2><strong><b>Decision Tree in Python With scikit learn<\/b><\/strong><\/h2>\n<p>This program has three parts. First it trains a small tree on the Iris flower dataset and prints it as readable rules. Then it uses the breast cancer dataset, with 569 patients and 30 measurements each, to show how depth and pruning change the result. Last it compares Gini with entropy. Both datasets come with scikit learn, so there is no file to load.<\/p>\n<div style=\"margin: 20px 0; border: 1px solid #d9e0ea; border-radius: 8px; overflow: hidden;\">\n<div style=\"background: #0f2a4a; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">python<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #f5f8fc; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">from sklearn.datasets import load_breast_cancer, load_iris\r\nfrom sklearn.model_selection import train_test_split\r\nfrom sklearn.tree import DecisionTreeClassifier, export_text\r\n\r\n# Part 1: a small tree you can read, on the Iris data\r\niris = load_iris()\r\nXi_train, Xi_test, yi_train, yi_test = train_test_split(\r\n    iris.data, iris.target, test_size=0.3, random_state=0, stratify=iris.target\r\n)\r\nsmall = DecisionTreeClassifier(max_depth=2, random_state=0).fit(Xi_train, yi_train)\r\nprint(export_text(small, feature_names=list(iris.feature_names)))\r\nprint(\"Iris test accuracy:\", round(small.score(Xi_test, yi_test), 3))\r\n\r\n# Part 2: depth, overfitting and pruning on the breast cancer data\r\ndata = load_breast_cancer()\r\nX_train, X_test, y_train, y_test = train_test_split(\r\n    data.data, data.target, test_size=0.3, random_state=42, stratify=data.target\r\n)\r\nprint(\"depth  train  test\")\r\nfor depth in [1, 2, 3, 5, 8, None]:\r\n    m = DecisionTreeClassifier(max_depth=depth, random_state=42).fit(X_train, y_train)\r\n    print(depth, round(m.score(X_train, y_train), 3), round(m.score(X_test, y_test), 3), \"leaves:\", m.get_n_leaves())\r\n\r\nprint(\"ccp_alpha  leaves  train  test\")\r\nfor alpha in [0.0, 0.005, 0.01, 0.02]:\r\n    m = DecisionTreeClassifier(ccp_alpha=alpha, random_state=42).fit(X_train, y_train)\r\n    print(alpha, m.get_n_leaves(), round(m.score(X_train, y_train), 3), round(m.score(X_test, y_test), 3))\r\n\r\n# Part 3: Gini against entropy\r\nfor criterion in [\"gini\", \"entropy\"]:\r\n    m = DecisionTreeClassifier(criterion=criterion, max_depth=3, random_state=42).fit(X_train, y_train)\r\n    print(criterion, round(m.score(X_test, y_test), 3))<\/pre>\n<div style=\"background: #1f2937; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">output<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #ffffff; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">|--- petal width (cm) &lt;= 0.80\r\n|   |--- class: 0\r\n|--- petal width (cm) &gt;  0.80\r\n|   |--- petal width (cm) &lt;= 1.75\r\n|   |   |--- class: 1\r\n|   |--- petal width (cm) &gt;  1.75\r\n|   |   |--- class: 2\r\n\r\nIris test accuracy: 0.956\r\ndepth  train  test\r\n1 0.927 0.912 leaves: 2\r\n2 0.965 0.918 leaves: 4\r\n3 0.98 0.924 leaves: 7\r\n5 0.995 0.93 leaves: 14\r\n8 1.0 0.918 leaves: 16\r\nNone 1.0 0.918 leaves: 16\r\nccp_alpha  leaves  train  test\r\n0.0 16 1.0 0.918\r\n0.005 7 0.985 0.93\r\n0.01 5 0.977 0.924\r\n0.02 3 0.957 0.918\r\ngini 0.924\r\nentropy 0.947<\/pre>\n<\/div>\n<h3><strong><b>Part 1: A Tree You Can Read<\/b><\/strong><\/h3>\n<div style=\"margin: 20px 0; border: 1px solid #d9e0ea; border-radius: 8px; overflow: hidden;\">\n<div style=\"background: #1f2937; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">output<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #ffffff; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">|--- petal width (cm) &lt;= 0.80\r\n|   |--- class: 0\r\n|--- petal width (cm) &gt;  0.80\r\n|   |--- petal width (cm) &lt;= 1.75\r\n|   |   |--- class: 1\r\n|   |--- petal width (cm) &gt;  1.75\r\n|   |   |--- class: 2\r\n\r\nIris test accuracy: 0.956<\/pre>\n<\/div>\n<p>Two questions about petal width are enough to separate three species. Anyone can follow these rules without knowing any statistics, which is the great strength of a tree. The test set has 45 flowers, so 0.956 means 43 correct.<\/p>\n<h3><strong><b>Part 2: Depth and Overfitting<\/b><\/strong><\/h3>\n<div style=\"margin: 20px 0; border: 1px solid #d9e0ea; border-radius: 8px; overflow: hidden;\">\n<div style=\"background: #1f2937; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">output<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #ffffff; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">depth  train  test\r\n1      0.927  0.912  leaves: 2\r\n2      0.965  0.918  leaves: 4\r\n3      0.98   0.924  leaves: 7\r\n5      0.995  0.93   leaves: 14\r\n8      1.0    0.918  leaves: 16\r\nNone   1.0    0.918  leaves: 16<\/pre>\n<\/div>\n<ul>\n<li><b><\/b><strong><b>Training accuracy keeps climbing. <\/b><\/strong>It reaches 1.0 at depth 8, which means the tree has memorised the training set.<\/li>\n<li><b><\/b><strong><b>Test accuracy stops improving. <\/b><\/strong>It peaks at 0.93 for depth 5, then falls back to 0.918.<\/li>\n<li><b><\/b><strong><b>Mind the small differences. <\/b><\/strong>The test set has 171 patients, and one patient is worth about 0.6 points. The gap between 0.918 and 0.93 is two patients, so treat it as a rough sign, not a precise answer.<\/li>\n<li><b><\/b><strong><b>The tree stopped growing at 16 leaves. <\/b><\/strong>Depth 8 and no limit gave the same tree, because every leaf was already pure.<\/li>\n<\/ul>\n<h3><strong><b>Part 3: Pruning<\/b><\/strong><\/h3>\n<div style=\"margin: 20px 0; border: 1px solid #d9e0ea; border-radius: 8px; overflow: hidden;\">\n<div style=\"background: #1f2937; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">output<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #ffffff; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">ccp_alpha  leaves  train  test\r\n0.0        16      1.0    0.918\r\n0.005      7       0.985  0.93\r\n0.01       5       0.977  0.924\r\n0.02       3       0.957  0.918<\/pre>\n<\/div>\n<p>A small amount of pruning, with ccp_alpha at 0.005, cut the tree from 16 leaves to 7 and raised test accuracy from 0.918 to 0.93. That is a simpler tree that does the same job or a little better. Prune too hard, as at 0.02, and the tree becomes too simple and the score drops.<\/p>\n<h3><strong><b>Part 4: Gini Against Entropy<\/b><\/strong><\/h3>\n<div style=\"margin: 20px 0; border: 1px solid #d9e0ea; border-radius: 8px; overflow: hidden;\">\n<div style=\"background: #1f2937; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">output<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #ffffff; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">gini 0.924\r\nentropy 0.947<\/pre>\n<\/div>\n<p>On this data and this split, entropy scored 0.947 and Gini 0.924, a difference of about 4 patients out of 171. That does not prove one is better. In practice the two usually pick very similar trees, so do not spend time on this choice before tuning depth and pruning.<\/p>\n<h2><strong><b>Advantages and Disadvantages of Decision Trees<\/b><\/strong><\/h2>\n<h3><strong><b>Advantages<\/b><\/strong><\/h3>\n<ul>\n<li>Easy to understand and explain, and you can draw them.<\/li>\n<li>Handle both classification and regression.<\/li>\n<li>Need little data preparation. They do not need scaling.<\/li>\n<li>Handle numbers and categories, and capture curved and interacting patterns.<\/li>\n<li>Fairly robust to outliers, because splits depend on order, not on how extreme a value is.<\/li>\n<\/ul>\n<h3><strong><b>Disadvantages<\/b><\/strong><\/h3>\n<ul>\n<li>Overfit easily if left to grow, which is the biggest weakness.<\/li>\n<li>A small change in the data can produce a very different tree.<\/li>\n<li>Predictions are steps, not smooth curves, which is a limit for regression.<\/li>\n<li>Biased toward features with many distinct values.<\/li>\n<li>Usually less accurate than random forests or boosting, which combine many trees.<\/li>\n<\/ul>\n<h2><strong><b>Where Decision Trees Are Used<\/b><\/strong><\/h2>\n<ul>\n<li><b><\/b><b><\/b>Approving or rejecting loan applications with rules that can be explained.<\/li>\n<li><b><\/b><b><\/b>Triage and diagnosis support.<\/li>\n<li><b><\/b><b><\/b>Deciding which customers to target.<\/li>\n<li><b><\/b><b><\/b>Spotting faults from machine readings.<\/li>\n<li><b><\/b><strong><b>Inside other models. <\/b><\/strong>Random forests and boosting are built from many trees.<\/li>\n<\/ul>\n<h2><strong><b>Common Mistakes to Avoid<\/b><\/strong><\/h2>\n<ul>\n<li><b><\/b><strong><b>Training and scoring on the same data. <\/b><\/strong>Always score on a held out test set.<\/li>\n<li><b><\/b><strong><b>Leaving the depth unlimited. <\/b><\/strong>Set max_depth or use pruning.<\/li>\n<li><b><\/b><strong><b>Trusting one tree.c<\/b><\/strong>Check with cross validation or move to a random forest.<\/li>\n<li><b><\/b><strong><b>Reading too much into a tiny example.c<\/b><\/strong>A tree trained on 10 rows teaches the method, not a real result.<\/li>\n<li><b><\/b><strong><b>Mixing up entropy and information gain.c<\/b><\/strong>Entropy is the uncertainty. Gain is how much a split reduces it.<\/li>\n<li><b><\/b><strong><b>Using a regression tree for a category or the reverse.c<\/b><\/strong>Match the tree type to the target.<\/li>\n<\/ul>\n<p>A decision tree turns data into a set of questions that anyone can follow. Learn how entropy, information gain and Gini choose each split, limit the depth or prune to avoid overfitting, and always judge the tree on data it has not seen. Once you are comfortable with one tree, move on to random forest and boosting, which use many trees to get higher accuracy.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Decision Tree in Machine Learning: Introduction With Examples Think about how a doctor narrows down a diagnosis. Is there a fever? If yes, is there a cough? If yes, is the oxygen level low? Each answer rules out some possibilities and points toward others, until a conclusion is left. A decision tree works the same [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[32,1696],"tags":[431,430,428,429],"class_list":["post-4617","post","type-post","status-publish","format-standard","hentry","category-machine-learning","category-machine-learning-modules","tag-decision-tree-example","tag-decision-tree-in-machine-learning","tag-decision-tree-introduction","tag-decision-tree-introduction-with-examples"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Decision Tree in Machine Learning With Python Examples<\/title>\n<meta name=\"description\" content=\"Learn how decision trees work with entropy, information gain and Gini, follow a worked example, and run tested Python code with pruning in scikit learn.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Decision Tree in Machine Learning With Python Examples\" \/>\n<meta property=\"og:description\" content=\"Learn how decision trees work with entropy, information gain and Gini, follow a worked example, and run tested Python code with pruning in scikit learn.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/\" \/>\n<meta property=\"og:site_name\" content=\"Prwatech\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prwatech.in\/\" \/>\n<meta property=\"article:published_time\" content=\"2026-10-18T10:02:44+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-10-19T14:07:34+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre2.png\" \/>\n\t<meta property=\"og:image:width\" content=\"864\" \/>\n\t<meta property=\"og:image:height\" content=\"616\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Prwatech\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Eduprwatech\" \/>\n<meta name=\"twitter:site\" content=\"@Eduprwatech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Prwatech\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/\",\"url\":\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/\",\"name\":\"Decision Tree in Machine Learning With Python Examples\",\"isPartOf\":{\"@id\":\"https:\/\/prwatech.in\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre2.png\",\"datePublished\":\"2026-10-18T10:02:44+00:00\",\"dateModified\":\"2026-10-19T14:07:34+00:00\",\"author\":{\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3\"},\"description\":\"Learn how decision trees work with entropy, information gain and Gini, follow a worked example, and run tested Python code with pruning in scikit learn.\",\"breadcrumb\":{\"@id\":\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/#primaryimage\",\"url\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre2.png\",\"contentUrl\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre2.png\",\"width\":864,\"height\":616},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/prwatech.in\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Decision Tree Introduction with Examples\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/prwatech.in\/blog\/#website\",\"url\":\"https:\/\/prwatech.in\/blog\/\",\"name\":\"Prwatech\",\"description\":\"Share Ideas, Start Something Good.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/prwatech.in\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3\",\"name\":\"Prwatech\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g\",\"caption\":\"Prwatech\"},\"url\":\"https:\/\/prwatech.in\/blog\/author\/prwatech123\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Decision Tree in Machine Learning With Python Examples","description":"Learn how decision trees work with entropy, information gain and Gini, follow a worked example, and run tested Python code with pruning in scikit learn.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/","og_locale":"en_US","og_type":"article","og_title":"Decision Tree in Machine Learning With Python Examples","og_description":"Learn how decision trees work with entropy, information gain and Gini, follow a worked example, and run tested Python code with pruning in scikit learn.","og_url":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/","og_site_name":"Prwatech","article_publisher":"https:\/\/www.facebook.com\/prwatech.in\/","article_published_time":"2026-10-18T10:02:44+00:00","article_modified_time":"2026-10-19T14:07:34+00:00","og_image":[{"width":864,"height":616,"url":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre2.png","type":"image\/png"}],"author":"Prwatech","twitter_card":"summary_large_image","twitter_creator":"@Eduprwatech","twitter_site":"@Eduprwatech","twitter_misc":{"Written by":"Prwatech","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/","url":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/","name":"Decision Tree in Machine Learning With Python Examples","isPartOf":{"@id":"https:\/\/prwatech.in\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/#primaryimage"},"image":{"@id":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/#primaryimage"},"thumbnailUrl":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre2.png","datePublished":"2026-10-18T10:02:44+00:00","dateModified":"2026-10-19T14:07:34+00:00","author":{"@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3"},"description":"Learn how decision trees work with entropy, information gain and Gini, follow a worked example, and run tested Python code with pruning in scikit learn.","breadcrumb":{"@id":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/#primaryimage","url":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre2.png","contentUrl":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2020\/01\/dectre2.png","width":864,"height":616},{"@type":"BreadcrumbList","@id":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/decision-tree-introduction\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/prwatech.in\/blog\/"},{"@type":"ListItem","position":2,"name":"Decision Tree Introduction with Examples"}]},{"@type":"WebSite","@id":"https:\/\/prwatech.in\/blog\/#website","url":"https:\/\/prwatech.in\/blog\/","name":"Prwatech","description":"Share Ideas, Start Something Good.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/prwatech.in\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3","name":"Prwatech","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g","caption":"Prwatech"},"url":"https:\/\/prwatech.in\/blog\/author\/prwatech123\/"}]}},"_links":{"self":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/4617","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/comments?post=4617"}],"version-history":[{"count":19,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/4617\/revisions"}],"predecessor-version":[{"id":12022,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/4617\/revisions\/12022"}],"wp:attachment":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/media?parent=4617"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/categories?post=4617"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/tags?post=4617"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}