HIVE-16997 data data_error ai_generated true

Hive partition column value silently overwritten to NULL during dynamic partition insert

ID: data/hive-partition-column-null-overwrite

Also available as: JSON · Markdown · 中文
82%Fix Rate
85%Confidence
1Evidence
2023-09-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Apache Hive 3.1.2 active — — —
Apache Hive 4.0.0-alpha-1 active — — —
CDH 6.3.2 active — — —

Root Cause

When using dynamic partitioning in Hive INSERT OVERWRITE, the partition column in the SELECT clause is misinterpreted as a regular column if the partition column name matches a column being inserted, causing the partition value to be overwritten to NULL.

generic

中文

在Hive INSERT OVERWRITE中使用动态分区时，如果SELECT子句中的分区列名与插入的列名匹配，分区列会被错误地解释为普通列，导致分区值被覆盖为NULL。

Official Documentation

https://issues.apache.org/jira/browse/HIVE-16997

Workarounds

  1. 92% success Explicitly list the partition column in the SELECT clause with a different alias or use static partition assignment in the INSERT statement. 
    Explicitly list the partition column in the SELECT clause with a different alias or use static partition assignment in the INSERT statement.
  2. 75% success Use INSERT INTO with explicit partition specification instead of INSERT OVERWRITE, then manually clean duplicates. 
    Use INSERT INTO with explicit partition specification instead of INSERT OVERWRITE, then manually clean duplicates.

中文步骤

  1. Explicitly list the partition column in the SELECT clause with a different alias or use static partition assignment in the INSERT statement.
  2. Use INSERT INTO with explicit partition specification instead of INSERT OVERWRITE, then manually clean duplicates.

Dead Ends

Common approaches that don't work:

  1. 95% fail

    Nonstrict mode bypasses validation but does not fix the column mapping, so data is still overwritten.

  2. 80% fail

    Hive's column resolution is based on names, not positions, so extra columns cause schema mismatch.