Data Cleaning Means Different Things in Data Science vs. Data Engineering

A data engineer reflecting on their PySpark and ETL pipeline work identified a fundamental mindset difference between how data scientists and data engineers approach data cleaning. In data science, cleaning is guided by the needs of a specific analysis or model, focusing on making data useful and appropriate for answering a particular question. Data engineering, by contrast, demands reliable, automated processing with considerations like schema consistency, failure handling, scalability, and data contracts for downstream systems. How issues like missing values are handled also differs: a data scientist might impute a missing field to preserve a training observation, while a pipeline engineer must decide whether a missing field should reject, quarantine, or pass through a record based on business rules. The key takeaway is that in data engineering, data quality expectations must be encoded directly into the system rather than relying on a person to manually inspect and judge the data each time.