10 Pandas Tricks That Saved My EDA Workflow

Introduction

After running hundreds of EDA sessions, I've developed a set of go-to pandas patterns that save real time. Here are the 10 I use most.

1. Profile Your Data Immediately

import pandas as pd

df = pd.read_csv("data.csv")

# Quick overview
print(df.info())          # dtypes and nulls
print(df.describe())      # stats
print(df.nunique())       # cardinality per column
print(df.isnull().sum())  # missing counts

2. .query() for Readable Filtering

# Instead of this mess:
result = df[(df['age'] > 25) & (df['city'] == 'Jakarta') & (df['score'] > 0.8)]

# Use .query() — much cleaner:
result = df.query("age > 25 and city == 'Jakarta' and score > 0.8")

3. Vectorized String Operations

# Never loop over strings — use .str accessor
df['email_domain'] = df['email'].str.split('@').str[1]
df['name_upper'] = df['name'].str.upper()
df['has_keyword'] = df['text'].str.contains('machine learning', case=False)

4. pd.cut() for Binning

# Convert continuous to categorical bins
df['age_group'] = pd.cut(
    df['age'],
    bins=[0, 18, 25, 35, 50, 100],
    labels=['<18', '18-25', '25-35', '35-50', '50+']
)

5. .assign() for Chaining

# Instead of mutating df in place, chain with .assign()
result = (
    df
    .assign(bmi=lambda x: x['weight'] / (x['height'] ** 2))
    .assign(bmi_category=lambda x: pd.cut(x['bmi'], bins=[0, 18.5, 25, 30, 100]))
    .query("age > 18")
)

6. Memory Optimization

def reduce_mem_usage(df):
    for col in df.columns:
        col_type = df[col].dtype
        if col_type != object:
            c_min, c_max = df[col].min(), df[col].max()
            if str(col_type)[:3] == 'int':
                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:
                    df[col] = df[col].astype(np.int8)
                # ... continue for int16, int32
            elif str(col_type)[:5] == 'float':
                df[col] = df[col].astype(np.float32)
    return df

7. groupby + agg in One Line

summary = (
    df.groupby('category')
    .agg(
        count=('id', 'count'),
        avg_score=('score', 'mean'),
        total_revenue=('revenue', 'sum'),
        max_value=('value', 'max')
    )
    .reset_index()
    .sort_values('total_revenue', ascending=False)
)

8. .pipe() for Custom Transformations

def remove_outliers(df, col, n_std=3):
    mean, std = df[col].mean(), df[col].std()
    return df[np.abs(df[col] - mean) <= n_std * std]

def normalize(df, col):
    df[col] = (df[col] - df[col].min()) / (df[col].max() - df[col].min())
    return df

# Clean pipeline
clean_df = (
    df
    .pipe(remove_outliers, 'salary')
    .pipe(normalize, 'salary')
)

9. pd.json_normalize() for Nested JSON

import json

# For APIs that return nested JSON
data = [{"user": {"name": "Indra", "age": 20}, "score": 95}]
flat = pd.json_normalize(data, sep='_')
# Result: user_name, user_age, score

10. Efficient CSV Reading

# Only load what you need
df = pd.read_csv(
    "huge_file.csv",
    usecols=['id', 'date', 'value'],  # only these columns
    parse_dates=['date'],              # parse dates automatically
    dtype={'id': 'int32'},            # specify dtypes upfront
    chunksize=100_000                  # for really large files
)

Summary

These 10 patterns cover 90% of what I do in EDA:

• Profile → Query → Transform → Aggregate → Visualize

The key insight: think in vectors, not loops. Pandas is built on NumPy arrays; operating on whole columns at once is orders of magnitude faster than Python loops.