Engineer builds rule-based tool to automate Spark and dbt config tuning decisions

A data engineer with over a decade of experience has built a pipeline configuration tool called OptiPipe to replace informal, person-dependent tuning knowledge on data teams. The tool reads pipeline configs alongside actual run metrics and applies seven explicit threshold-based rules to flag issues such as over-provisioned memory, poor shuffle partition sizing, and inefficient dbt thread counts. Rather than using a machine learning model, the developer chose auditable rule logic so that every recommendation can be traced back to a specific condition and metric. All suggestions require human approval before any production config is changed, with no automated writes permitted. The tool currently offers full dbt integration via run_results.json uploads, while Spark support works on sample data pending a live cluster reader; a public demo is available at opti-pipe.onrender.com without sign-up.