Exploratory data analysis is the step most analysts rush through so they can get to modeling or dashboards. It's also the step that decides whether the rest of the work holds up. This guide walks through what to check, what to plot, and the small mistakes that quietly poison later stages.
Why EDA is the step people underestimate
Every dataset you inherit lies to you a little. Column names promise things the values don't deliver. Nulls sit where zeros should be. A "date" column contains six different formats and one row where somebody typed "yesterday". Exploratory data analysis is where you find these problems before they cost you a week of confused modeling.
Most junior analysts I've worked with treat EDA as a warm-up. You open the file, run df.head(), notice nothing horrifying, and move on to the interesting work: the regression, the dashboard, the forecast. The trouble is that the interesting work runs happily on broken assumptions. The chart still renders. The model still converges. The number at the bottom of the deck is still a number. It's just wrong, and you won't know it until someone in the meeting asks a question you can't answer.
Good EDA is boring, patient, and mostly about looking at the data one column at a time until you actually understand what you're holding.
Start with the shape
Before any plots, get the boring facts:
- How many rows and columns are there?
- What's the data type of each column, and does it match what the column is supposed to represent?
- How many rows are missing at least one value? Which columns leak the most nulls?
- Are there duplicate rows? Duplicate keys?
- If there's a date column, what's the time range, and are there gaps?
In pandas that's roughly df.shape, df.info(), df.isna().sum(), df.duplicated().sum(). In Excel it's a series of pivots and COUNTBLANK checks. In SQL it's COUNT(*), COUNT(DISTINCT id), and WHERE col IS NULL. The tool doesn't matter. The questions do.
A short example. Say you've been handed a year of sales data from a Kenyan appliance retailer. First look tells you there are 42,317 rows (one row per transaction) and thirteen columns. store_id should be an integer but it's stored as text, which means you can't sort it numerically without a cast. sale_date is text too, half of it as 2024-05-14 and half as 14/05/2024. Nothing has broken yet. But if you had gone straight into a monthly trend chart, half your data would have quietly landed in the wrong bucket.
The three passes
Once the shape is clear, EDA usually moves in three passes.
| Pass | What you're looking at | Typical output |
|---|---|---|
| Univariate | One column at a time | Histograms, box plots, value counts, summary stats |
| Bivariate | Two columns together | Scatter plots, grouped box plots, cross-tabs, correlations |
| Multivariate | Three or more together | Faceted plots, correlation heatmaps, pair plots, PCA |
You don't have to be religious about the order, but skipping straight to multivariate is how you end up staring at a correlation heatmap that means nothing because half the columns are corrupted.
Univariate: what does each column actually look like?
For numeric columns: min, max, mean, median, standard deviation, and a histogram or box plot. Real distributions are almost never a clean bell. Prices are usually right-skewed. Ages cluster in bands. Counts pile up near zero and have a long thin tail. The point of looking is to answer one question: does this distribution match what I'd expect for this variable, and if not, why not?
For categorical columns: value counts. Look at the top values, look at the tail. A retailer's product_category should have maybe fifteen values, not eighty. If you see eighty, somebody typed "Fridges", "fridges", "Fridge ", "Fridges " and the system stored them separately. That's a cleaning job, not a modeling job.
Bivariate: which columns move together?
Now you're asking the interesting questions. Does the average basket size go up on weekends? Do certain stores skew toward high-margin products? How does the price of a refrigerator relate to its energy rating?
Scatter plots for two numeric variables. Grouped box plots when one variable is categorical. Cross-tabs when both are categorical. Pearson correlations are useful but easy to misread. They only capture linear relationships, and a correlation of 0.02 doesn't mean the variables are unrelated. It means a straight line doesn't describe them.
Multivariate: what's the wider structure?
Correlation heatmaps, pair plots, and small-multiples charts are your tools here. The question is whether the patterns you spotted in bivariate still hold when you condition on a third variable. Sometimes the "effect" you thought you saw between price and sales disappears once you split by store region. That's an important finding, and you would have missed it staying at the bivariate level.
Missing values and outliers, without the panic
Two topics dominate every EDA tutorial and both get handled badly. Missing values first.
The question that matters isn't "how do I fill them?" but "why are they missing?" A missing customer_phone on a walk-in retail transaction means the customer didn't share it. A missing discount_percent might mean zero discount, or it might mean the field was added to the system halfway through the year. These are different problems. The first is fine to leave as null or code as "not provided". The second means half your data is structurally different from the other half, and you need to know the cutover date before you do anything else.
Outliers get the same treatment. A value of 999,999 in a sale_amount column might be a legitimate bulk sale to a hotel, or it might be a sentinel value somebody used to mean "unknown". You cannot tell from the number alone. You need to look at the row, and often two or three rows around it in the source system, before you decide.
Tools that pull their weight
A short opinion on tooling. For most EDA work I reach for:
- pandas for the actual manipulation. Nothing else gives you the same speed for reshape, group-by, and quick summary stats.
- matplotlib and seaborn for plotting. Plotly is nicer for interactive dashboards, but for exploration you want plots you can generate fast and throw away.
ydata-profiling(formerly pandas-profiling) as a first-pass shortcut. It generates a full report with distributions, missing value patterns, and correlations in a few lines. Don't use it as a substitute for looking at the data. Use it to tell you where to look first.- Excel for anything under about 100,000 rows where you want to poke around visually. Sorting, filtering, and conditional formatting on a real spreadsheet still teaches you things a notebook can hide.
If you're working directly against a warehouse (BigQuery, Snowflake, Postgres), most of the univariate work is SELECT COUNT(*), COUNT(DISTINCT col), MIN(col), MAX(col), AVG(col) FROM t GROUP BY .... You don't get pretty plots, but you can get the facts.
Mistakes I keep seeing
A few patterns show up over and over in reviews:
- Averaging over things that shouldn't be averaged. A mean price across products of very different types is a number that describes nothing real. Segment first, then average.
- Treating IDs as numeric.
store_idvalues of 12, 15, and 34 have no meaningful order. Don't put them on a numeric axis or feed them into a regression as-is. - Ignoring the sample. If your data covers only Nairobi stores, don't quietly generalize to Kenya. Note the scope in your findings and keep noting it.
- Stopping too early. Once you find one interesting pattern, it's tempting to write it up and move on. Half the time a fifth or sixth plot would have shown that the pattern only exists because of a data quality issue upstream.
- Only looking at what the brief asked for. The brief says "understand basket size". You spend an hour on basket size and never notice that a third of the rows have a negative quantity, which is how the system codes returns. That third of the rows is now sitting inside every "sales" number you'll ever produce off this dataset.
The tell of good EDA isn't a pretty notebook. It's whether, at the end, you can sit across from someone and answer questions about the data without hedging.
Frequently asked questions
What's the difference between EDA and data cleaning?
They overlap and usually happen in the same sitting. EDA is the process of asking questions about the data (what's here, how is it distributed, what relates to what). Data cleaning is what you do when EDA surfaces a problem: recoded categories, wrong data types, missing values, duplicated rows. You rarely finish one before starting the other. In practice you loop between them.
Do I need Python for EDA, or can I do it in Excel?
For datasets under about 100,000 rows, Excel is fine and often faster for the first look because you can sort, filter, and eyeball values directly. Beyond that, pandas in Python or a SQL warehouse becomes necessary because the file stops opening in reasonable time. The mental process is identical either way. The tool follows the size of the data.
How long should EDA take?
There's no clean answer, but as a rough guide: on a project where the whole timeline is two weeks, expect to spend the first three to five days almost entirely in EDA. If you finish faster than that, you probably skipped something. If the dataset is unfamiliar or messy, budget more. It's the phase where cutting corners costs the most later.
Is EDA still worth doing if I'm just building a dashboard?
Yes, and arguably more so. A model that's fed bad data will at least sometimes fail to converge or produce obviously strange outputs. A dashboard will happily display wrong numbers in a nicely designed card and nobody will notice for months. If you're building reporting, spend real time on EDA first so you know exactly what each number in your dashboard is counting.