_edited_edited.png)
Suze Orman once said, “Cleanliness is a state of purity, clarity, and precision. ” This not only applies to our day-to-day cleaning, but it also applies to our data analysis. Just as we keep our house clean and tidy for our comfort, the data we use for analysis needs to be cleaned before we draw our insights. Just learn data cleaning as cleaning our own home.
What is Data Cleaning?
Data Cleaning is the process of making the data we are going to analyze free from unwanted, incorrect, duplicate, or incomplete data. Data cleaning is almost like cleaning our house, which is a necessary task to ensure that our space is organized, usable, and ready for guests. Just like we wouldn’t like to be over to a cluttered, dirty house, we shouldn’t use messy data for analysis. Data cleaning is to make sure the information we’re working on is consistent and easy to work with.
Importance of Data Cleaning?
Let’s imagine we invited a new friend to our house for dinner, but instead of a clean, welcoming space, we have all things out of place. That would be such a bad experience for our poor new friend. In the same way, messy data can draw wrong conclusions and can make misinformed decisions. So, Data cleaning ensures that our data is polished and ready to use for accurate analysis.
Key Steps in Data Cleaning
Although there is no one absolute way to prescribe the exact steps in data cleaning, it is crucial to establish a template for our data cleaning process so that we know we are doing it the right way every time. However, the processes will vary from dataset to dataset. Below are the few steps we follow for most of the datasets.
Handling Missing Data is just like Finding Lost Items
Sometimes, we lose our socks and car keys, but we manage this by either using a spare key or the phone key or by finding another pair of socks. Similarly, missing data can be handled by first figuring out what’s missing and determining how to fill the gap. So, missing data can be either removed or can be filled with other values.
Removing the Missing Items: We remove the data with missing values, if the missing data is minor and won’t have much effect on our results, it’s like deciding to leave that missing sock behind and just going with the other pair. In large datasets where removing a few rows of data due to missing data doesn’t impact the analysis.
Filling the Missing Gaps: Just as we might use a spare key to replace the one we lost, we can fill missing data with estimates by using mean, median, mode, or predictive algorithms to impute values.
2. Removing Duplicates, like Getting Rid of Clutter
Think of a day when you were cleaning your drawer and found five phone chargers, three of which you no longer have devices. These are like duplicates that take up the space unwantedly and create a mess. Sorting them, just keeping what we actually need, creates more organized space.
In the same analogy, those extra chargers were repeated records that would serve no purpose. Removing them from our dataset makes it clean and streamlined.
3. Handling Outliers is Like That One Oversized Item That Doesn’t Belong
Let’s imagine our neatly arranged living room with one oversized armchair that’s too big for that space. It sticks out so awkwardly that it almost disrupts the flow of the room. This is like an outlier in our data, whose value is noticeably different from the rest.
In data cleaning, we identify and assess these outliers to decide whether they add value, like a statement piece in the room, or we can remove and replace, like replacing the oversized chair with one that fits better. So, to replace the data, we have :
IQR Method: This method is used to calculate the interquartile range (IQR) and identify outliers as points falling below Q1 - 1.5IQR or above Q3 + 1.5IQR just as using a measuring tape to decide if the chair is truly too large for the space or not.
Z-Score: This gives us how many standard deviations a data point is away from the mean. So, values with very high absolute Z-scores are considered outliers. In the same way as the chair is so big that it blocks the doorway, it’s definitely a problem. Z-scores help you identify extreme outliers that deviate significantly from the average.
4. Standardizing Data is identical to Organizing Items into Proper Places
While tidying our home, it’s important to organize things into proper categories, books go into the bookshelf, dishes in the cupboard, shoes into the closet, and so on. It is more like data standardization, which is about organizing data so that it’s consistent. For example, if we have dates in multiple formats (DD/MM/YYYY and MM/DD/YYYY), it’s like trying to organize two kinds of clothes in the same drawer we need to standardize the data format to make it easy for our analysis.
This case is the same for measurements in different units like pounds and kilograms, we need to convert them to the same unit, just like organizing all your shoes in one place.
5. Correcting Inconsistent Data just like Matching Socks in the Laundry
One would never fold unmatched socks, like one blue and another green socks. To keep them consistent, we pair them correctly and remove all the mismatches. Similarly, inconsistent data values do not fall in line with standard analysis. Data cleaning has to be done to remove the inconsistency in data. Hence, the data will turn out to be consistent and analytically easy. For example, in cases where one row's spelling of a city is NY, and another is spelled out as "New York," we take either to avoid confusion and bring clarity into the dataset.
6. Encoding Categorical Data just as Organizing by Labels
Let’s say we want to sort our clothes by type, such as shirts, pants, dresses, etc. We then label each pile so that we know where to find each item when we need them. Similarly, in data analysis, categorical data is encoded. Encoding is transforming non-numeric categories like gender, country names into numbers or specific codes so that they can be processed by the algorithms for visualization and other analyses. This is almost like labeling those piles of clothes for easy access.
Cleaning Supplies for Data Cleaning
Just as we use a vacuum, broom, or cleaning spray to tidy up our home, there are tools that help streamline the data cleaning process:
Excel/Google Sheets: These are very familiar cleaning tools used for smaller datasets. We can remove duplicates, find and replace inconsistent data, and even fill in missing values manually.
Python (Pandas): Python’s Pandas library is just like having a best cleaning kit that can handle large, messy datasets. With simple commands, we can filter, fill, and correct data inconsistencies.
SQL: As our data is mostly stored in a relational database, SQL helps us to clean our data by running queries to remove duplicates, filter out missing data, and make transformations.
Best Housekeeping Tips for Data Cleaning
To ensure your data is mess-free, here are some of the best housekeeping practices:
Always Keep a Backup: Before we repaint a rented house to our color preference, we save the old paint swatches and take note of the original color so that it’s useful while moving to another house so that we can revert back to the same color of paint to avoid fines. Similarly, we always maintain a backup of the original dataset, just in case anything goes wrong in the cleaning process, we can always revert back .
Clean in Phases: Just as cleaning the house is easier when going room by room. In the data cleaning process, we also go step by step to removing duplicates first, then handling missing data, and so on.
Documenting the Process: Just as we note down where we store our seasonal items, documenting our cleaning process helps to understand what steps we followed before when needed.
Validating our Results: Taking one final look before the guests arrive to make sure everything is right in place. Check whether everything is in order or not with our dataset and whether our data is ready for analysis.
Conclusion:
Just as a clean home provides a welcoming environment for our guests, clean data creates a solid foundation for accurate analysis. By following these steps of data cleaning and using the right tools, we can ensure our data is organized, accurate, and ready to be transformed into valuable insights. So, let’s think of data cleaning as your data's version of tidying up our space, which is essential, ongoing, and much needed.
For purity, clarity, and precision ! Happy cleaning!