Parallel aggregation of dataframes
Dec 5, 2019··
1 min read
Georg Heiler
Spark is known for its great API and fairly easy to build widely scalable data processing jobs.
However, it is also known to be hard to fine tune.
One trick might be handy for you: parallel aggregations. As the resilient distributed datasets (RDD) backing data frames are immutable it can make a lot of sense to operate on them in parallel if your cluster has enough resources (free slots).
To do so start a spark shell:
spark-shell
and execute:
val df = Seq((1, "A"), (2, "A"), (3, "B")).toDF("value", "group")
df.show
+-----+-----+
|value|group|
+-----+-----+
| 1| A|
| 2| A|
| 3| B|
+-----+-----+
Assuming you have several aggregations you want to perform - they are a good candidate to be faster in parallel.
val aggregations = Seq(1,2,3)
aggregations.par.foreach(a => {
println(s"starting ${a}")
df.filter('value === a).groupBy("value").count.write.parquet(s"result.parquet/group=${a}")
println(s"completing ${a}")
})
The depicted aggregations are a bit too simplistic - but I think you understand the usefulness of this method.
If you truly want to proces the aggregations in parallel do not forget to set
--conf spark.scheduler.mode='FAIR'
. Otherwise tasks are submitted in parallel, but are handled in a FiFo queue. This still makes sense though if you have enough slots.summary
Using parallel operations certain jobs can be sped up in spark and made much faster!