Course
Pandas - Data Analysis with Python
103 lessons across 10 modules
Beginner to intermediate, assuming Core Python and basic NumPy. Reading data from files, databases and APIs, selecting and filtering, cleaning, transforming, grouping, joining, strings and dates, reshaping, and working at scale, then preparing data for machine learning - ending in four projects. The road is laid out in full; lessons are being written one at a time.
Pandas Fundamentals
Series, DataFrames, and a first look at real data
Prerequisite: the NumPy course →What is Pandas?
1Labelled, tabular data in Python - and where it sits beside NumPy.
A spreadsheet task done in three lines of Pandas
Installing and Importing Pandas
2Installing with pip, importing as pd, and checking the version.
import pandas as pd, the convention every example follows
Pandas Data Structures
3Series, DataFrame, and the Index that labels both.
One column pulled out of a DataFrame as a Series
Creating a Series
4A labelled one-dimensional array, with a default or custom index.
Scores indexed by student name instead of 0, 1, 2
Creating a DataFrame
5From dictionaries, lists of rows, and NumPy arrays.
Names, ages, and salaries as a three-column table
DataFrame Attributes
6shape, size, columns, index, dtypes, and ndim.
A salary column stored as text, spotted from dtypes
Inspecting Data
7head, tail, sample, info, and describe - a first look at any dataset.
The five commands to run on every new file
Reading and Writing Data
Getting data in from files, databases, and APIs - and back out
Reading CSV Files
8read_csv, and the defaults it applies without asking.
A leading-zero postcode turned into a number
Writing CSV Files
9to_csv, and index=False to avoid an extra column.
An unnamed index column appearing on every re-read
Reading Excel Files
10read_excel, sheets, and the engine it needs installed.
Reading one named sheet from a workbook
Writing Excel Files
11to_excel, and writing several sheets into one workbook.
A report with one sheet per region
Reading JSON
12read_json, and json_normalize for nested records.
Nested API records flattened into columns
Reading SQL Data
13read_sql with a connection, and filtering in SQL rather than Pandas.
A query that returns a thousand rows instead of a million
Reading Data from APIs
14Fetching JSON over HTTP and turning it into a DataFrame.
A paginated API collected into one table
Handling File Paths and Encodings
15Paths that work everywhere, and encoding errors on read.
A UnicodeDecodeError fixed by naming the encoding
Important read_*() Options
16usecols, dtype, skiprows, nrows, na_values, and parse_dates.
A large file read in a fraction of the memory with usecols and dtype
Selecting and Filtering Data
Getting exactly the rows and columns you need
Selecting Columns
17One column as a Series, a list of columns as a DataFrame.
df["name"] against df[["name"]], and why the brackets matter
Selecting Rows
18Row slicing with plain brackets, and why it confuses people.
df[0:3] selecting rows while df["x"] selects a column
loc[]
19Selection by label - and loc slices include the end.
loc[0:2] returning three rows, not two
iloc[]
20Selection by position, with the end excluded like Python.
The first three rows and first two columns by position
Boolean Filtering
21Keeping the rows where a condition is true.
Every employee earning over 50,000
Multiple Conditions
22& and | with brackets - never and or or.
A ValueError from and, fixed with & and brackets
isin()
23Matching any value from a list.
Rows from three chosen departments
between()
24An inclusive range filter.
Ages from 25 to 35
query()
25Filters written as a readable string.
A three-condition filter that reads like a sentence
Data Cleaning
The work that takes most of the time in any real analysis
Understanding Missing Data
26NaN, None, NA, and NaT - and why missing is not zero.
An average that changes completely once missing values are handled
Detecting Missing Values
27isna and isnull, and counting the gaps per column.
A column that is forty per cent missing
Removing Missing Values
28dropna, with how, subset, and thresh.
Dropping only the rows missing a required field
Filling Missing Values
29fillna with a constant, a statistic, or per-column values.
Missing ages filled with the median, not the mean
Forward Fill and Backward Fill
30Carrying the last known value forward or the next one back.
Gaps in a sensor reading filled from the reading before
Handling Duplicate Data
31Finding and removing duplicates, and choosing which to keep.
Keeping the latest record for each customer
Handling Incorrect Data
32Values that are present but impossible.
A negative age and a date in the future, corrected
Data Type Conversion
33astype, to_numeric with errors, and nullable integer types.
astype(int) failing on a NaN, fixed with Int64
Cleaning Column Names
34Lowercase, stripped, and consistent names you can type.
Column names with trailing spaces breaking a lookup
Handling Outliers
35Detecting with IQR or z-scores, then deciding what to do.
An outlier that was a real value, not an error
Data Cleaning Workflow
36Missing values, duplicates, invalid values, types, and outliers, in order.
A raw file turned into a clean dataset, step by step
Sorting, Ranking and Data Transformation
Ordering data and deriving new columns from it
Sorting Data
37sort_values, ascending or descending, and where NaN ends up.
The highest salaries first
Sorting by Multiple Columns
38Several sort keys, each with its own direction.
By department ascending, then salary descending
sort_index()
39Restoring or imposing order by the index.
A filtered frame put back into its original order
Ranking Data
40rank, and the tie methods that change the answer.
Two tied salaries ranked average, min, and dense
Creating New Columns
41Derived columns - and the chained-assignment warning.
A SettingWithCopyWarning from a filtered frame, and the fix
Applying Functions
42apply on a Series and on rows - flexible, but slow.
An apply that a vectorised expression replaces
map()
43Mapping values through a dictionary or a function.
Department codes turned into department names
replace()
44Swapping specific values, including with patterns.
Every "N/A" and "-" replaced with a real missing value
assign()
45Adding columns inside a method chain.
A clean, readable chain of transformations
Conditional Transformation
46Values chosen by condition - np.where and np.select over apply.
High, medium, and low salary bands in one vectorised call
Grouping and Aggregation
Split, apply, combine - the core of analysis
Understanding groupby()
47Split into groups, apply a function, combine the results.
Employees split by department, drawn out
Aggregation
48Reducing each group to a single value.
Average salary per department
Multiple Aggregations
49Several statistics per group with agg.
Mean, min, and max salary in one table
count(), sum(), mean()
50The everyday aggregates, and count against size with NaN.
count and size disagreeing on a column with gaps
min() and max()
51Extremes per group, and idxmin and idxmax to find the row.
The highest-paid person in each department, not just the salary
Grouping by Multiple Columns
52Groups for every combination, and the MultiIndex they return.
Revenue per region per product
Named Aggregations
53Choosing the output column names as you aggregate.
total_revenue and avg_order instead of generated names
value_counts()
54How often each value occurs, as counts or proportions.
The share of orders from each region
pivot_table()
55A quick summary table - grouping laid out as rows and columns.
Revenue with regions down the side and months across the top
Practical Grouping Problems
56Real questions answered with groupby.
Total revenue, average revenue, and quantity for every region
Combining DataFrames
Stacking tables and joining related ones
Joins in the SQL course →Why DataFrames Need to Be Combined
57Real data arrives in several tables linked by keys.
Customers in one file, orders in another
concat()
58Stacking frames with the same columns on top of each other.
Twelve monthly files combined into one year
Horizontal Concatenation
59Placing frames side by side, aligned on the index.
Misaligned indexes producing a wall of NaN
merge()
60SQL-style joins on key columns.
Customers and orders joined on customer_id
Inner Join
61Only the keys found in both frames.
Customers who have placed at least one order
Left Join
62Every row from the left, matched where possible.
Every customer, with NaN where there are no orders
Right Join
63Every row from the right - usually rewritten as a left join.
The same result as a right and as a left join
Outer Join
64Every key from both sides, and the indicator column.
Records that exist in only one of two systems
Merge on Multiple Columns
65Composite keys, and validate to catch duplicate keys.
A merge that silently multiplied rows, caught by validate
join()
66Joining on the index rather than a column.
Two frames indexed by date joined in one call
combine_first()
67Filling the gaps in one frame from another.
Missing contact details patched from a second source
Working with Strings, Dates and Time
Text columns, datetimes, and time series
String Operations
68The .str accessor for vectorised text operations.
Every name trimmed and lowercased in one line
Searching Strings
69contains, startswith, and case and missing-value handling.
A contains that crashed on NaN until na=False was set
Splitting Strings
70Splitting text into columns with expand.
A full name split into first and last name columns
Replacing Strings
71Replacing text, with and without regular expressions.
Currency symbols stripped so a price column can be numeric
Converting Strings to Dates
72to_datetime, explicit formats, and day-first ambiguity.
03/04/2024 read as March when it meant April
Extracting Date Components
73Year, month, day, weekday, and more through .dt.
Orders grouped by day of the week
Date Filtering
74Filtering by date range, and partial date strings.
Every order from the first quarter
Date Differences
75Subtracting dates to get durations.
Days between order and delivery
Resampling Time-Series Data
76Changing frequency - daily into weekly or monthly.
Daily sales totalled into monthly sales
Time-Series Basics
77A datetime index, and shifting values in time.
Each day compared with the same day last week
Advanced Pandas
Reshaping, window functions, and working at scale
MultiIndex
78Creating an index with more than one level.
Rows labelled by both region and year
Hierarchical Indexing
79Selecting from a MultiIndex - tuples, xs, and slices.
Every year for one region, then one year across all regions
pivot()
80Reshaping long data to wide - it fails on duplicate entries.
A ValueError from pivot on a repeated pair
pivot_table()
81The reshaping view - aggregating duplicates, margins, and fill values.
The pivot that failed, fixed with pivot_table and a total row
melt()
82Reshaping wide data to long.
A column per month turned into one row per month
stack() and unstack()
83Moving index levels between rows and columns.
A grouped result unstacked into a readable grid
Window Functions
84rolling, expanding, and ewm, and what each window covers.
A seven-day, a cumulative, and a weighted average side by side
Rolling Average
85Smoothing noisy data, with min_periods and centring.
A seven-day average of daily sales
Efficient Pandas Operations
86What makes Pandas code fast or slow.
An analysis rewritten to run twenty times faster
Vectorization vs apply()
87Whole-column operations against a Python function per row.
An apply timed against the vectorised equivalent
Memory Optimization
88Smaller numeric types and the category dtype.
A DataFrame cut to a quarter of its memory
Large Dataset Processing
89When data no longer fits comfortably in memory, and the options.
Reading only the columns and rows you need
Chunk Processing
90Reading a file in pieces and combining the results.
A file larger than memory aggregated chunk by chunk
Pandas for Data Science and Projects
Exploring data, preparing it for models, and four projects
Exploratory Data Analysis
91A structured first pass to learn what a dataset contains.
An EDA checklist run on an unfamiliar dataset
Descriptive Statistics
92Using summary statistics to answer real questions.
Whether the average salary misleads, answered with the median
Correlation
93How columns move together - and that correlation is not causation.
A strong correlation with an obvious hidden cause
Detecting Data Patterns
94Trends, seasonality, and segments in the data.
A weekly pattern hidden inside daily sales
Pandas and NumPy
95Moving between DataFrames and arrays, and when to drop down.
to_numpy for a calculation Pandas makes awkward
Pandas and Matplotlib
96Plotting straight from a DataFrame.
A monthly revenue chart from one grouped frame
Pandas and Scikit-learn
97DataFrames as model input, and keeping column names through a pipeline.
Features from a DataFrame passed into a model
Preparing Data for Machine Learning
98Cleaning, encoding, and selecting features for a model.
Categorical columns one-hot encoded with get_dummies
End-to-End Data Analysis Workflow
99Read, inspect, clean, transform, analyse, and report.
A question taken all the way from raw file to answer
Project 1 - Student Performance Analysis
100Cleaning, filtering, grouping, ranking, and performance analysis.
The subjects and students that need attention, identified
Project 2 - E-Commerce Sales Analysis
101Revenue by region and product, monthly sales, and order value.
Top products and customers, and the average order value
Project 3 - Employee Data Analysis
102Salary by department, experience, distribution, and hiring trends.
Joining trends over time and salary spread per department
Final Project - Real-World Data Analysis Pipeline
103A public dataset taken from source to charts and a final report.
A clean dataset, a notebook, charts, and business insights