Pandas 性能优化：从分钟到秒的提速技巧

# Pandas 性能优化：从分钟到秒的提速技巧 ## 1. 数据类型优化 ```python import pandas as pd import numpy as np df = pd.read_csv("large_data.csv") print(df.memory_usage(deep=True).sum() / 1024**2, "MB") # 优化数值类型 def optimize_dtypes(df): for col in df.select_dtypes(include=["int64"]).columns: col_min = df[col].min() col_max = df[col].max() if col_min >= 0: if col_max -128 and col_max -32768 and col_max 1000 else "low") # 快：np.where df["category"] = np.where(df["amount"] > 1000, "high", "low") # 更复杂的条件用 np.select conditions = [ df["amount"] > 10000, df["amount"] > 1000, df["amount"] > 0 ] choices = ["premium", "high", "standard"] df["tier"] = np.select(conditions, choices, default="free") ``` ## 3. eval 和 query ```python # 标准写法（创建多个临时 Series） result = df[(df["age"] > 25) & (df["salary"] > 50000) & (df["dept"] == "engineering")] # eval 写法（内存效率更高） result = df.query("age > 25 and salary > 50000 and dept == 'engineering'") # 复杂计算 df.eval("bonus = salary * 0.1 + performance_score * 500", inplace=True) ``` ## 4. 分块处理大文件 ```python chunks = pd.read_csv("huge_file.csv", chunksize=100000) results = [] for chunk in chunks: processed = chunk.groupby("category")["amount"].sum() results.append(processed) final = pd.concat(results).groupby(level=0).sum() ``` ## 5. Categorical 类型 ```python # 字符串列重复值多时使用 Categorical df["country"] = df["country"].astype("category") # 内存减少 90% 以上，分组操作加速 5-10 倍 ``` 这些优化技巧组合使用可以让数据处理速度提升一到两个数量级。对于超大规模数据，建议直接迁移到 Polars 或 DuckDB。