{"id":2862,"date":"2026-10-18T17:27:55","date_gmt":"2026-10-18T17:27:55","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=2862"},"modified":"2026-10-19T14:49:08","modified_gmt":"2026-10-19T14:49:08","slug":"data-cleaning-in-machine-learning","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/","title":{"rendered":"Data Cleaning in Machine Learning"},"content":{"rendered":"<h1><strong><b>Data Cleaning in Machine Learning<\/b><\/strong><\/h1>\n<p>Imagine a shop owner who keeps sales in a notebook. One page says 12 kg of rice, another says twelve, a third says 12 kilos, and one line just says na. Before anyone can add up the week&#8217;s sales, someone has to read every line and make it consistent. That tidying is data cleaning, and it is the least glamorous and most important part of machine learning.<\/p>\n<p>A model can only learn from what it is given. If the data has duplicates, blanks, typing errors, and the same city spelled three ways, the model learns the mess. Practitioners often say that most of a project goes into preparing data, and your own experience will soon show why. Good cleaning rarely makes a headline, yet it often raises accuracy more than switching to a fancier algorithm.<\/p>\n<p>In this tutorial, we explain what data cleaning is and why it matters, then follow the steps in order: removing unwanted rows and columns, handling missing values, fixing structural errors and managing outliers. Everything is shown on one small, messy table with tested pandas code and the real output of each step. We also cover a point that most tutorials skip: how to fill missing values without leaking information from your test data. We finish with a checklist, common mistakes and an FAQ.<\/p>\n<h2><strong><b>What is Data Cleaning?<\/b><\/strong><\/h2>\n<p>Data cleaning is the process of finding and fixing problems in a dataset so that it is accurate, consistent and ready for analysis or modelling. The problems include missing values, duplicate rows, wrong entries, inconsistent labels, wrong data types and extreme values. It is also called data cleansing or data scrubbing, and it is the first part of data preprocessing.<\/p>\n<p>The right steps depend on the dataset. A table of customer records and a set of sensor readings are broken in different ways. There is no single recipe, but the steps below cover the problems that come up most often.<\/p>\n<h2><strong><b>Why does Data Cleaning Matter?<\/b><\/strong><\/h2>\n<ul>\n<li><b><\/b><strong><b>Models learn what they see. <\/b><\/strong>Errors in the data become errors in the predictions.<\/li>\n<li><b><\/b><strong><b>Many tools fail on blanks. <\/b><\/strong>Several algorithms stop with an error if any value is missing.<\/li>\n<li><b><\/b><strong><b>Inconsistent labels split the groups. <\/b><\/strong>Berkeley and BERKELEY look like two different places to a computer.<\/li>\n<li><b><\/b><strong><b>Decisions rest on the results. <\/b><\/strong>A wrong average based on dirty data can lead to a wrong business decision.<\/li>\n<\/ul>\n<h2><strong><b>Data Cleaning Steps in Machine Learning<\/b><\/strong><\/h2>\n<p>Follow these steps. We will go through each one with code. In the program, the structural fixes run before the gaps are filled, and the note under Step 3 explains why.<\/p>\n<ol>\n<li><b><\/b><strong><b>Look at the data <\/b><\/strong>and count what is missing.<\/li>\n<li><b><\/b><strong><b>Remove unwanted observations, <\/b><\/strong>which means duplicate rows and rows you do not need.<\/li>\n<li><b><\/b><strong><b>Handle missing values<\/b><\/strong>by filling or removing them.<\/li>\n<li><b><\/b><strong><b>Fix structural errors, <\/b><\/strong>such as inconsistent labels, wrong types and placeholder text.<\/li>\n<li><b><\/b><strong><b>Manage outliers <\/b><\/strong>by checking whether they are errors or real.<\/li>\n<li><b><\/b><strong><b>Check the result <\/b><\/strong>and keep a note of what you changed.<\/li>\n<\/ol>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4668\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/datacln1.png\" alt=\"machine learning approach to data cleaning\" width=\"850\" height=\"591\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/datacln1.png 986w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/datacln1-300x209.png 300w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/datacln1-768x534.png 768w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<h2><strong><b>Our Example Data<\/b><\/strong><\/h2>\n<p>We will clean a small table of 10 properties, with the columns PID, ST_NUM, ST_NAME, OWN_OCCUPIED, NUM_BEDROOMS, NUM_BATH and SQ_FT. I built it to contain the problems you will meet in real life. The values are made up. It has placeholder text such as na and double dashes, a duplicate row, a word in a number column, and a street spelt two ways. The full program is below, and each step is explained after it.<\/p>\n<div style=\"margin: 20px 0; border: 1px solid #d9e0ea; border-radius: 8px; overflow: hidden;\">\n<div style=\"background: #0f2a4a; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">python<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #f5f8fc; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">import pandas as pd\r\nfrom io import StringIO\r\nfrom sklearn.impute import SimpleImputer\r\nfrom sklearn.model_selection import train_test_split\r\n\r\nraw = StringIO(\"\"\"PID,ST_NUM,ST_NAME,OWN_OCCUPIED,NUM_BEDROOMS,NUM_BATH,SQ_FT\r\n100001,104,PUTNAM,Y,3,1,1000\r\n100002,197,LEXINGTON,N,3,1.5,--\r\n100003,,LEXINGTON,N,n\/a,1,850\r\n100004,201,BERKELEY,12,1,NA,700\r\n100004,201,BERKELEY,12,1,NA,700\r\n100005,203,Berkeley,Y,3,2,1600\r\n100006,207,BERKELEY,Y,na,1,800\r\n100007,,WASHINGTON,,2,HURLEY,950\r\n100008,213,TREMONT,Y,1,1,\r\n100009,215,TREMONT,Y,na,2,1800\r\n\"\"\")\r\n\r\n# Step 1: load, and tell pandas which placeholder texts mean missing\r\ndf = pd.read_csv(raw, na_values=[\"na\", \"n\/a\", \"NA\", \"--\"])\r\nprint(\"Rows and columns:\", df.shape)\r\nprint(df.isnull().sum())\r\n\r\n# Step 2: remove exact duplicate rows\r\nprint(\"Duplicates:\", int(df.duplicated().sum()))\r\ndf = df.drop_duplicates()\r\n\r\n# Step 3: fix text in a numeric column, then check the types\r\nprint(df[\"NUM_BATH\"].dtype)\r\ndf[\"NUM_BATH\"] = pd.to_numeric(df[\"NUM_BATH\"], errors=\"coerce\")\r\nprint(df[\"NUM_BATH\"].dtype)\r\n\r\n# Step 4: fix inconsistent labels\r\ndf[\"ST_NAME\"] = df[\"ST_NAME\"].str.upper().str.strip()\r\nprint(sorted(df[\"ST_NAME\"].unique()))\r\n\r\n# Step 5: OWN_OCCUPIED should only hold Y or N, anything else is an error\r\ndf[\"OWN_OCCUPIED\"] = df[\"OWN_OCCUPIED\"].where(df[\"OWN_OCCUPIED\"].isin([\"Y\", \"N\"]))\r\nprint(df[\"OWN_OCCUPIED\"].value_counts(dropna=False).to_dict())\r\n\r\n# Step 6: fill the gaps, using the median for measures and the most common value for labels\r\n# Identifiers such as PID and ST_NUM are left alone, because a median street number means nothing\r\ndf[\"NUM_BEDROOMS\"] = df[\"NUM_BEDROOMS\"].fillna(round(df[\"NUM_BEDROOMS\"].median()))\r\nfor col in [\"NUM_BATH\", \"SQ_FT\"]:\r\n    df[col] = df[col].fillna(df[col].median())\r\ndf[\"OWN_OCCUPIED\"] = df[\"OWN_OCCUPIED\"].fillna(df[\"OWN_OCCUPIED\"].mode()[0])\r\nprint(df.isnull().sum().to_dict())\r\nprint(df)\r\n\r\n# Step 7: the safe way for machine learning, learn the fill values from training data only\r\nX = pd.DataFrame({\"sq_ft\": [1000, 850, None, 700, 1600, 800, 950, None, 1800, 1200]})\r\nX_train, X_test = train_test_split(X, test_size=0.3, random_state=0)\r\nimputer = SimpleImputer(strategy=\"median\").fit(X_train)\r\nprint(\"Median learned from training data:\", float(imputer.statistics_[0]))<\/pre>\n<div style=\"background: #1f2937; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">output<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #ffffff; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">Rows and columns: (10, 7)\r\nPID             0\r\nST_NUM          2\r\nST_NAME         0\r\nOWN_OCCUPIED    1\r\nNUM_BEDROOMS    3\r\nNUM_BATH        2\r\nSQ_FT           2\r\ndtype: int64\r\nDuplicates: 1\r\nstr\r\nfloat64\r\n['BERKELEY', 'LEXINGTON', 'PUTNAM', 'TREMONT', 'WASHINGTON']\r\n{'Y': 5, 'N': 2, nan: 2}\r\n{'PID': 0, 'ST_NUM': 2, 'ST_NAME': 0, 'OWN_OCCUPIED': 0, 'NUM_BEDROOMS': 0, 'NUM_BATH': 0, 'SQ_FT': 0}\r\n      PID  ST_NUM     ST_NAME OWN_OCCUPIED  NUM_BEDROOMS  NUM_BATH   SQ_FT\r\n0  100001   104.0      PUTNAM            Y           3.0       1.0  1000.0\r\n1  100002   197.0   LEXINGTON            N           3.0       1.5   950.0\r\n2  100003     NaN   LEXINGTON            N           2.0       1.0   850.0\r\n3  100004   201.0    BERKELEY            Y           1.0       1.0   700.0\r\n5  100005   203.0    BERKELEY            Y           3.0       2.0  1600.0\r\n6  100006   207.0    BERKELEY            Y           2.0       1.0   800.0\r\n7  100007     NaN  WASHINGTON            Y           2.0       1.0   950.0\r\n8  100008   213.0     TREMONT            Y           1.0       1.0   950.0\r\n9  100009   215.0     TREMONT            Y           2.0       2.0  1800.0\r\nMedian learned from training data: 900.0<\/pre>\n<\/div>\n<h3><strong><b>Step 1: Look at the Data and Count What is Missing<\/b><\/strong><\/h3>\n<p>Start by loading the file and counting the blanks in each column. Note that pandas will not treat text such as na, NA or a double dash as missing unless you say so. The na_values setting tells pandas which texts mean missing.<\/p>\n<div style=\"margin: 20px 0; border: 1px solid #d9e0ea; border-radius: 8px; overflow: hidden;\">\n<div style=\"background: #1f2937; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">output<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #ffffff; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">Rows and columns: (10, 7)\r\nPID             0\r\nST_NUM          2\r\nST_NAME         0\r\nOWN_OCCUPIED    1\r\nNUM_BEDROOMS    3\r\nNUM_BATH        2\r\nSQ_FT           2<\/pre>\n<\/div>\n<p>The original page uses isnull().any(), which only returns True or False for each column. Use isnull().sum() to see how many are missing, since 2 blanks in 10 rows is very different from 2 in 10,000.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4671\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning2.png\" alt=\"Data Cleaning in Machine Learning - Removing Unwanted Observations\" width=\"850\" height=\"440\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning2.png 718w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning2-300x155.png 300w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<h2><strong><b>Step 2: Remove Unwanted Observations<\/b><\/strong><\/h2>\n<p>Observations are rows. Unwanted ones include exact duplicates and rows that are not relevant to your question. In the example, row 100004 appears twice.<\/p>\n<div style=\"margin: 20px 0; border: 1px solid #d9e0ea; border-radius: 8px; overflow: hidden;\">\n<div style=\"background: #1f2937; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">output<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #ffffff; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">Duplicates: 1<\/pre>\n<\/div>\n<p>Dropping duplicates with drop_duplicates() leaves 9 rows. Dropping a column is a different job, and it belongs to feature selection. You can drop columns that have no use, such as a building name, with drop and axis set to 1.<\/p>\n<p><strong>Be Careful With dropna()<\/strong><\/p>\n<p>dropna() removes every row that has even one missing value. It does not remove only the rows with the most blanks, as the original page says. On our small table, it would delete most of the rows. Use it only when very few rows are affected. To remove rows with too many blanks, set the thresh option, which keeps only rows that have at least that many real values.<\/p>\n<h2><strong><b>Step 3: Handle Missing Values<\/b><\/strong><\/h2>\n<p>You can remove rows with blanks, or fill the blanks with a sensible value. Removing is fine when very few rows are affected. Otherwise, fill them. The common choices are these:<\/p>\n<div style=\"border: 1px solid #d9e0ea; border-radius: 8px; overflow-x: auto; margin: 20px 0; -webkit-overflow-scrolling: touch;\">\n<table style=\"width: 100%; min-width: 700px; border-collapse: collapse; font-family: Arial,Helvetica,sans-serif; font-size: 16px; line-height: 1.6; color: #1f2937;\">\n<caption style=\"caption-side: top; text-align: left; padding: 14px 16px; font-weight: bold; font-size: 17px; color: #0f2a4a;\">Ways to fill missing values<\/caption>\n<thead>\n<tr>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38; width: 210px;\" scope=\"col\">Method<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38;\" scope=\"col\">Use it for<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38;\" scope=\"col\">Watch out for<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">Median<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Numbers, especially skewed ones such as area or income<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Hides real variation<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">Mean<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Numbers that are bell shaped<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Pulled by outliers<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">Mode, the most common value<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Labels such as Y or N<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Can over represent the majority<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a;\" scope=\"row\">Interpolation<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Ordered data such as time series<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Meaningless for IDs<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; color: #0f2a4a;\" scope=\"row\">Leave blank, add a flag<\/th>\n<td style=\"padding: 12px 16px;\">When missing itself carries meaning<\/td>\n<td style=\"padding: 12px 16px;\">Needs a model that can handle blanks<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4672\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning3.png\" alt=\"Data Cleaning in Machine Learning - Missing Data Handling\" width=\"850\" height=\"444\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning3.png 700w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning3-300x157.png 300w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<p>In the program, the structural fixes in Step 4 run first, because a column that still holds text cannot be filled with a median. The page follows the order of the original article, so read Steps 3 and 4 together. Bedrooms get the median rounded to a whole number, bathrooms and area get the median, and OWN_OCCUPIED gets the most common value. The median of bedrooms is 2.5, which rounds to 2. I did not fill PID or ST_NUM. A median street number or a median ID number means nothing, so those stay as they are. The original page interpolates PID, which has no meaning.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4673\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning4.png\" alt=\" machine learning approach to data cleaning - Missing Data Handling Result\" width=\"850\" height=\"469\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning4.png 717w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning4-300x166.png 300w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<div style=\"margin: 20px 0; border: 1px solid #d9e0ea; border-radius: 8px; overflow: hidden;\">\n<div style=\"background: #1f2937; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">output<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #ffffff; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">{'PID': 0, 'ST_NUM': 2, 'ST_NAME': 0, 'OWN_OCCUPIED': 0, 'NUM_BEDROOMS': 0, 'NUM_BATH': 0, 'SQ_FT': 0}<\/pre>\n<\/div>\n<p>Every column that should be filled is complete. Only ST_NUM still has 2 blanks, on purpose. A column of identifiers is better left blank than filled with a made up value.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4674\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning-5.png\" alt=\" data cleaning in machine learning python - Missing Data Handling Result\" width=\"850\" height=\"433\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning-5.png 723w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning-5-300x153.png 300w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<h3><strong><b>Fill Without Leaking: Use SimpleImputer<\/b><\/strong><\/h3>\n<p>There is one important trap. If you calculate the median from the whole dataset and then split it into train and test, information from the test rows has flowed into the training data. This is called data leakage, and it makes scores look better than they will be in real use. The safe method is to learn the fill value from the training data only, and then apply it to both parts. SimpleImputer does exactly that:<\/p>\n<div style=\"margin: 20px 0; border: 1px solid #d9e0ea; border-radius: 8px; overflow: hidden;\">\n<div style=\"background: #1f2937; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">output<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #ffffff; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">Median learned from training data: 900.0<\/pre>\n<\/div>\n<p>The median of 900 was learned from the training rows alone. Use it with fit on the training data and transform on the test data, and put it in a pipeline so it happens automatically.<\/p>\n<h2><strong><b>Step 4: Fix Structural Errors<\/b><\/strong><\/h2>\n<p>Structural errors are inconsistencies in how the data is written. They come from typing, combining files, or different people entering data.<\/p>\n<h3><strong><b>Inconsistent Labels<\/b><\/strong><\/h3>\n<p>In our table, one street is written BERKELEY in some rows and Berkeley in another. Making the case consistent with str.upper() and removing extra spaces with str.strip() solves this.<\/p>\n<div style=\"margin: 20px 0; border: 1px solid #d9e0ea; border-radius: 8px; overflow: hidden;\">\n<div style=\"background: #1f2937; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">output<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #ffffff; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">['BERKELEY', 'LEXINGTON', 'PUTNAM', 'TREMONT', 'WASHINGTON']<\/pre>\n<\/div>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4675\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning-6.png\" alt=\"Data Cleaning in Machine Learning - Structural Error Solving\" width=\"850\" height=\"456\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning-6.png 718w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning-6-300x161.png 300w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<h3><strong><b>Text in a Number Column<\/b><\/strong><\/h3>\n<p>The NUM_BATH column holds the word HURLEY in one row, which makes pandas read the whole column as text. The simple fix is pd.to_numeric with errors set to coerce, which turns anything that is not a number into a missing value. That is shorter and safer than the loop in the original page, which assumed the index runs from 0 to n minus 1 and breaks after rows are dropped.<\/p>\n<div style=\"margin: 20px 0; border: 1px solid #d9e0ea; border-radius: 8px; overflow: hidden;\">\n<div style=\"background: #1f2937; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">output<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #ffffff; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">str\r\nfloat64<\/pre>\n<\/div>\n<p>The first line shows the type before the fix, which is text. Older pandas versions call it object. The second line shows the type after the fix, which is a decimal number. Check data types with df.dtypes before and after.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4676\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning-7.png\" alt=\"Data Cleaning in Machine Learning - Structural Error Solving Result\" width=\"850\" height=\"455\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning-7.png 718w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning-7-300x160.png 300w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<h3><strong><b>Values That do Not Belong<\/b><\/strong><\/h3>\n<p>OWN_OCCUPIED should hold only Y or N, but two rows hold the number 12. A simple rule works well: keep the value only if it is Y or N, and otherwise treat it as missing. This is the reverse of the loop on the original page, which blanked the numbers it should have kept.<\/p>\n<div style=\"margin: 20px 0; border: 1px solid #d9e0ea; border-radius: 8px; overflow: hidden;\">\n<div style=\"background: #1f2937; color: #ffffff; font-family: Arial,Helvetica,sans-serif; font-size: 14px; font-weight: bold; padding: 8px 16px;\">output<\/div>\n<pre style=\"margin: 0; padding: 16px; background: #ffffff; color: #1f2937; font-family: 'Courier New',Courier,monospace; font-size: 14px; line-height: 1.7; overflow-x: auto; white-space: pre;\">{'Y': 5, 'N': 2, nan: 2}<\/pre>\n<\/div>\n<p>The 2 missing values here are one that was already blank and one that came from the bad value 12.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4677\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning8.png\" alt=\"machine learning approach to data cleaning - Structural Error Solving\" width=\"850\" height=\"447\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning8.png 698w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning8-300x158.png 300w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<h3><strong><b>Bathrooms are Not Always Whole Numbers<\/b><\/strong><\/h3>\n<p>The original page converts NUM_BATH to an integer. That is wrong, since half baths exist, and our table has a value of 1.5. Keep it as a decimal number. If you need whole numbers in a column that has blanks, use the nullable Int64 type, because the ordinary integer type fails when a blank remains.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4678\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning-9.png\" alt=\"data cleaning in machine learning python - Structural Error Solving Result\" width=\"850\" height=\"431\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning-9.png 722w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/Data-cleaning-9-300x152.png 300w\" sizes=\"auto, (max-width: 850px) 100vw, 850px\" \/><\/p>\n<p>Here is the cleaned table after Steps 2, 3 and 4:<\/p>\n<div style=\"border: 1px solid #d9e0ea; border-radius: 8px; overflow-x: auto; margin: 20px 0; -webkit-overflow-scrolling: touch;\">\n<table style=\"width: 100%; min-width: 700px; border-collapse: collapse; font-family: Arial,Helvetica,sans-serif; font-size: 16px; line-height: 1.6; color: #1f2937;\">\n<caption style=\"caption-side: top; text-align: left; padding: 14px 16px; font-weight: bold; font-size: 17px; color: #0f2a4a;\">Sample housing data<\/caption>\n<thead>\n<tr>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38; white-space: nowrap;\" scope=\"col\">PID<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38; white-space: nowrap;\" scope=\"col\">ST_NUM<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38; white-space: nowrap;\" scope=\"col\">ST_NAME<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38; white-space: nowrap;\" scope=\"col\">OWN_OCCUPIED<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38; white-space: nowrap;\" scope=\"col\">NUM_BEDROOMS<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38; white-space: nowrap;\" scope=\"col\">NUM_BATH<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38; white-space: nowrap;\" scope=\"col\">SQ_FT<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">100001<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">104<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">PUTNAM<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Y<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">3<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">1.0<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">1000<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">100002<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">197<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">LEXINGTON<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">N<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">3<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">1.5<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">950<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">100003<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">blank<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">LEXINGTON<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">N<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">2<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">1.0<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">850<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">100004<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">201<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">BERKELEY<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Y<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">1<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">1.0<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">700<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">100005<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">203<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">BERKELEY<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Y<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">3<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">2.0<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">1600<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">100006<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">207<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">BERKELEY<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Y<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">2<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">1.0<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">800<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">100007<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">blank<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">WASHINGTON<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Y<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">2<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">1.0<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">950<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">100008<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">213<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">TREMONT<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Y<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">1<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">1.0<\/td>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">950<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">100009<\/th>\n<td style=\"padding: 12px 16px;\">215<\/td>\n<td style=\"padding: 12px 16px;\">TREMONT<\/td>\n<td style=\"padding: 12px 16px;\">Y<\/td>\n<td style=\"padding: 12px 16px;\">2<\/td>\n<td style=\"padding: 12px 16px;\">2.0<\/td>\n<td style=\"padding: 12px 16px;\">1800<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>&nbsp;<\/p>\n<p>The 10 rows became 9 after the duplicate was removed. The SQ_FT for property 100008 was blank. It is now 950, the median of the other areas.<\/p>\n<h2><strong><b>Step 5: Manage Outliers<\/b><\/strong><\/h2>\n<p>Outliers are values far from the rest. They are not always wrong. A very large flat can be real, while a flat of 1 square foot is surely a typing error. The right approach is to check the cause before changing anything. Keep real values, fix clear errors, and compare results with and without suspicious points. Do not remove them automatically. Our full guide to outliers shows how to detect them with box plots, the z score and the IQR rule, and why blind removal can throw away your most important rows.<\/p>\n<h2><strong><b>A Data Cleaning Checklist<\/b><\/strong><\/h2>\n<div style=\"border: 1px solid #d9e0ea; border-radius: 8px; overflow-x: auto; margin: 20px 0; -webkit-overflow-scrolling: touch;\">\n<table style=\"width: 100%; min-width: 600px; border-collapse: collapse; font-family: Arial,Helvetica,sans-serif; font-size: 16px; line-height: 1.6; color: #1f2937;\">\n<caption style=\"caption-side: top; text-align: left; padding: 14px 16px; font-weight: bold; font-size: 17px; color: #0f2a4a;\">Data cleaning checklist<\/caption>\n<thead>\n<tr>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38; width: 220px;\" scope=\"col\">Check<\/th>\n<th style=\"background: #0f2a4a; color: #ffffff; text-align: left; padding: 12px 16px; border-bottom: 2px solid #0b1f38;\" scope=\"col\">What to look for<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Shape and types<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Do the columns have the right types? Use <code style=\"font-family: 'Courier New',Courier,monospace; font-size: 15px; background: #eef2f7; padding: 2px 6px; border-radius: 4px;\">df.dtypes<\/code><\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Missing values<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Count them per column with <code style=\"font-family: 'Courier New',Courier,monospace; font-size: 15px; background: #eef2f7; padding: 2px 6px; border-radius: 4px;\">isnull().sum()<\/code><\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Placeholder text<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">na, NA, a dash, a blank string, 999 or 0 used for unknown<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Duplicates<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Exact duplicate rows, and near duplicates<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Labels<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Spelling, case and spaces in text columns<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Valid values<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Ages below zero, a Y or N column holding numbers<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Outliers<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Check the cause before changing anything<\/td>\n<\/tr>\n<tr style=\"background: #f5f8fc;\">\n<th style=\"text-align: left; padding: 12px 16px; border-bottom: 1px solid #e5e9f0; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Leakage<\/th>\n<td style=\"padding: 12px 16px; border-bottom: 1px solid #e5e9f0;\">Learn fill values from training data only<\/td>\n<\/tr>\n<tr style=\"background: #ffffff;\">\n<th style=\"text-align: left; padding: 12px 16px; color: #0f2a4a; white-space: nowrap;\" scope=\"row\">Record of changes<\/th>\n<td style=\"padding: 12px 16px;\">Write down what you removed and why<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>&nbsp;<\/p>\n<h2><strong><b>Common Mistakes to Avoid<\/b><\/strong><\/h2>\n<ul>\n<li><b><\/b><strong><b>Missing the hidden blanks. <\/b><\/strong>Text like na, dashes and 999 hide as real values unless you list them in na_values.<\/li>\n<li><b><\/b><strong><b>Filling before splitting. <\/b><\/strong>It leaks test information into training. Fit imputers on training data only.<\/li>\n<li><b><\/b><strong><b>Filling identifiers. <\/b><\/strong>A median PID or street number is meaningless.<\/li>\n<li><b><\/b><strong><b>Dropping every row with a blank. <\/b><\/strong>You may lose most of your data.<\/li>\n<li><b><\/b><strong><b>Forcing integers. <\/b><\/strong>Bathrooms can be 1.5.<\/li>\n<li><b><\/b><strong><b>Cleaning by hand in a spreadsheet. <\/b><\/strong>You cannot repeat it. Write the steps as code.<\/li>\n<li><b><\/b><strong><b>Not keeping a record. <\/b><\/strong>You will not remember what you changed and why.<\/li>\n<li><b><\/b><strong><b>Removing outliers automatically. <\/b><\/strong>Check the cause first.<\/li>\n<\/ul>\n<p>Data cleaning is not exciting, but it is where good models begin. Work through the steps in order, write them as code so they can be repeated, learn fill values from training data only, and keep a note of every change. Run the program above on your own messy file, count the problems at each step, and you will see how much of your accuracy was hiding in the data all along.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Data Cleaning in Machine Learning Imagine a shop owner who keeps sales in a notebook. One page says 12 kg of rice, another says twelve, a third says 12 kilos, and one line just says na. Before anyone can add up the week&#8217;s sales, someone has to read every line and make it consistent. That [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[32,1696],"tags":[410,409],"class_list":["post-2862","post","type-post","status-publish","format-standard","hentry","category-machine-learning","category-machine-learning-modules","tag-data-cleaning-in-machine-learning","tag-data-cleaning-steps-in-machine-learning"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Mastering Data Cleaning &amp; Data Preprocessing<\/title>\n<meta name=\"description\" content=\"Explore Data Cleaning Steps in Machine Learning and learn how to clean data for analysis by Data Cleaning Guide from Prwatech today.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Mastering Data Cleaning &amp; Data Preprocessing\" \/>\n<meta property=\"og:description\" content=\"Explore Data Cleaning Steps in Machine Learning and learn how to clean data for analysis by Data Cleaning Guide from Prwatech today.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/\" \/>\n<meta property=\"og:site_name\" content=\"Prwatech\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prwatech.in\/\" \/>\n<meta property=\"article:published_time\" content=\"2026-10-18T17:27:55+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-10-19T14:49:08+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/datacln1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"986\" \/>\n\t<meta property=\"og:image:height\" content=\"686\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Prwatech\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Eduprwatech\" \/>\n<meta name=\"twitter:site\" content=\"@Eduprwatech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Prwatech\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/\",\"url\":\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/\",\"name\":\"Mastering Data Cleaning & Data Preprocessing\",\"isPartOf\":{\"@id\":\"https:\/\/prwatech.in\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/datacln1.png\",\"datePublished\":\"2026-10-18T17:27:55+00:00\",\"dateModified\":\"2026-10-19T14:49:08+00:00\",\"author\":{\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3\"},\"description\":\"Explore Data Cleaning Steps in Machine Learning and learn how to clean data for analysis by Data Cleaning Guide from Prwatech today.\",\"breadcrumb\":{\"@id\":\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/#primaryimage\",\"url\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/datacln1.png\",\"contentUrl\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/datacln1.png\",\"width\":986,\"height\":686},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/prwatech.in\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Data Cleaning in Machine Learning\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/prwatech.in\/blog\/#website\",\"url\":\"https:\/\/prwatech.in\/blog\/\",\"name\":\"Prwatech\",\"description\":\"Share Ideas, Start Something Good.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/prwatech.in\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3\",\"name\":\"Prwatech\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g\",\"caption\":\"Prwatech\"},\"url\":\"https:\/\/prwatech.in\/blog\/author\/prwatech123\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Mastering Data Cleaning & Data Preprocessing","description":"Explore Data Cleaning Steps in Machine Learning and learn how to clean data for analysis by Data Cleaning Guide from Prwatech today.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/","og_locale":"en_US","og_type":"article","og_title":"Mastering Data Cleaning & Data Preprocessing","og_description":"Explore Data Cleaning Steps in Machine Learning and learn how to clean data for analysis by Data Cleaning Guide from Prwatech today.","og_url":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/","og_site_name":"Prwatech","article_publisher":"https:\/\/www.facebook.com\/prwatech.in\/","article_published_time":"2026-10-18T17:27:55+00:00","article_modified_time":"2026-10-19T14:49:08+00:00","og_image":[{"width":986,"height":686,"url":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/datacln1.png","type":"image\/png"}],"author":"Prwatech","twitter_card":"summary_large_image","twitter_creator":"@Eduprwatech","twitter_site":"@Eduprwatech","twitter_misc":{"Written by":"Prwatech","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/","url":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/","name":"Mastering Data Cleaning & Data Preprocessing","isPartOf":{"@id":"https:\/\/prwatech.in\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/#primaryimage"},"image":{"@id":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/#primaryimage"},"thumbnailUrl":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/datacln1.png","datePublished":"2026-10-18T17:27:55+00:00","dateModified":"2026-10-19T14:49:08+00:00","author":{"@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3"},"description":"Explore Data Cleaning Steps in Machine Learning and learn how to clean data for analysis by Data Cleaning Guide from Prwatech today.","breadcrumb":{"@id":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/#primaryimage","url":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/datacln1.png","contentUrl":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/09\/datacln1.png","width":986,"height":686},{"@type":"BreadcrumbList","@id":"https:\/\/prwatech.in\/blog\/machine-learning\/machine-learning-modules\/data-cleaning-in-machine-learning\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/prwatech.in\/blog\/"},{"@type":"ListItem","position":2,"name":"Data Cleaning in Machine Learning"}]},{"@type":"WebSite","@id":"https:\/\/prwatech.in\/blog\/#website","url":"https:\/\/prwatech.in\/blog\/","name":"Prwatech","description":"Share Ideas, Start Something Good.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/prwatech.in\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3","name":"Prwatech","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g","caption":"Prwatech"},"url":"https:\/\/prwatech.in\/blog\/author\/prwatech123\/"}]}},"_links":{"self":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/2862","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/comments?post=2862"}],"version-history":[{"count":22,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/2862\/revisions"}],"predecessor-version":[{"id":12028,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/2862\/revisions\/12028"}],"wp:attachment":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/media?parent=2862"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/categories?post=2862"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/tags?post=2862"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}