communication observability ai_generated true

Datadog custom metric bill spike — tag cardinality explosion

ID: communication/datadog-custom-metric-cardinality-explosion

Also available as: JSON · Markdown
82%Fix Rate
87%Confidence
4Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —

Root Cause

High-cardinality tags (user_id) on custom metrics create millions of unique time series.

generic

Workarounds

  1. 92% success Audit tags for cardinality: remove unbounded tags like user_id 
    Replace user_id with user_tier in Metrics Summary

    Sources: https://docs.datadoghq.com/metrics/custom_metrics/

  2. 85% success Use Metrics without Limits to control indexing 
    Configure tag allowlist: only index low-cardinality tags

    Sources: https://docs.datadoghq.com/metrics/metrics-without-limits/

Dead Ends

Common approaches that don't work:

  1. Just remove all tags from metrics 80% fail

    Removing all tags makes metrics ungroupable and useless for debugging

  2. Switch to logs for everything instead of metrics 70% fail

    Log ingestion at high volume is often more expensive than metrics