The original title is: "Data quality bugs in a 37M-row predictive maintenance dataset: from 11 rows to 36 million"

Short summary

A predictive maintenance model on 37M telemetry records from an iron ore mine revealed four silent data quality bugs ranging from 11 rows (UTF-8 encoding corruption) to 36M rows (literal string 'NULL' instead of null values). The author demonstrates how sampling hides problems affecting 0.6% of data and advocates for fix-reporting dataclasses that quantify what each cleaning step touches. The key lesson: stop writing cleaning code first, and instead make every fix report what it touched.

• •Four silent data quality bugs in 37M rows ranged from 11 to 36M affected rows
• •Literal string 'NULL' fooled isna() checks, making a 97% empty column report as 100% populated
• •Sampling hid a 237K-row problem; full-dataset runs and fix-reporting dataclasses are essential

Generated with AI, which can make mistakes.