EAV Pipeline Design Lets Engineers Add New Data Sources Without Changing Code

A software engineer published a mini data engineering project demonstrating how to unify multiple wide-format CSV files from different manufacturing sources into a standardized Entity-Attribute-Value (EAV) format. The core problem addressed is that identical metrics — such as production quantity or temperature — arrive under different column names and units across files from different plants and vendors. Instead of writing source-specific conditional logic in pipeline code, the approach externalizes all column mappings and unit conversions into JSON configuration files. The pipeline then transforms raw CSVs into a long EAV format and aggregates them into a gold metric mart, with idempotency ensured via file content hashing. A key test confirms that onboarding a new data source requires only adding one config file, with no changes to the pipeline code itself.