Apache Iceberg in Production: Compaction, Catalogs, and the Pitfalls Nobody Warns You About

Short summary

Apache Iceberg simplifies data lake management but requires three production disciplines: automate file compaction from day one to prevent micro-batch file explosion that degrades query speed 40x, use hidden partitioning for transparent partition pruning without schema column hacks, and schedule weekly snapshot expiration to control metadata bloat and S3 costs. Choose your catalog strategy early (AWS Glue for single-cloud, Nessie/REST for multi-engine, Unity for Databricks) because migrations are costly.

• •Automate file compaction on day one targeting 128–512MB per file to prevent micro-batch explosion and query slowdown
• •Use Iceberg's hidden partitioning to automatically prune reads without coupling query logic to physical layout
• •Schedule snapshot expiration and orphan file cleanup weekly to prevent metadata bloat and runaway S3 costs

Generated with AI, which can make mistakes.