How Salting Technique Fixed a 40-Minute PySpark Data Skew Bottleneck

A PySpark job processing a large transactions-to-customer join was completing 95% of its tasks in under 4 minutes, but one or two straggler tasks were taking over 40 minutes to finish. The Spark UI revealed the cause: data skew, where a handful of high-volume customer IDs held disproportionately large row counts, forcing all their data into a single partition processed by one executor. A pre-join aggregation on the join key confirmed that a small number of customer IDs had row counts orders of magnitude higher than the rest. The fix applied was a technique called salting, which appends a random value to the skewed join key to split one large partition into several smaller ones distributed across multiple executors. After implementing salting with 10 salt buckets, the workload was rebalanced across the cluster and the job's overall runtime returned to acceptable levels.