# 读取到pandas时Parquet十进制精度溢出——值被截断或转换为NaN

- **ID:** `data/parquet-decimal-precision-overflow-pandas`
- **领域:** data
- **类别:** type_error
- **验证级别:** ai_generated
- **修复率:** 85%

## 根因

Parquet文件存储高精度十进制数（如DECIMAL(38,10)），但pandas使用Python的float64或int64，无法表示如此高的精度，导致溢出或静默截断。

## 版本兼容性

| 版本 | 状态 | 引入 | 弃用 |
|------|------|------|------|
| pandas 2.2.0 | active | — | — |
| pyarrow 15.0.0 | active | — | — |
| Apache Parquet 1.13.0 | active | — | — |

## 解决方案

1. ```
   Read the Parquet file with pyarrow directly and convert to pandas using `pq.read_table(path).to_pandas(timestamp_as_object=True)` which converts decimals to Python Decimal objects. Example: `import pyarrow.parquet as pq; table = pq.read_table('data.parquet'); df = table.to_pandas(timestamp_as_object=True, date_as_object=True)`
   ```
2. ```
   Use `pd.read_parquet(path, engine='pyarrow', dtype_backend='numpy_nullable')` and then manually convert decimal columns to `pd.StringDtype()` to preserve precision: `df['dec_col'] = df['dec_col'].astype('string')`
   ```
3. ```
   If the data fits within 38 digits, use `pd.read_parquet(path, engine='pyarrow', use_nullable_dtypes=True)` which uses `pd.ArrowDtype` for decimals, preserving precision as Python Decimal objects
   ```

## 无效尝试

- **** — This uses pyarrow for conversion but still may overflow if the decimal precision exceeds 38 digits or if pyarrow's default decimal type cannot map to pandas. (70% 失败率)
- **** — Float64 can only represent ~15-17 significant digits; high-precision decimals will be truncated or rounded, losing data. (90% 失败率)
- **** — Fastparquet has similar limitations and may silently convert decimals to float64, causing the same overflow. (75% 失败率)